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Instructions for Completing Consolidated State Application 

Accountability Workbook 

By January 31 , 2003, States must complete and submit to the Department this 
Consolidated State Application Accountability Workbook. We understand that some of 
the critical elements for the key principles may still be under consideration and may not 
yet be final State policy by the January 31 due date. States that do not have final 
approval for some of these elements or that have not finalized a decision on these 
elements by January 31 should, when completing the Workbook, indicate the status of 
each element which is not yet official State policy and provide the anticipated date by 
which the proposed policy will become effective. In each of these cases, States must 
include a timeline of steps to complete to ensure that such elements are in place by 
May 1 , 2003, and implemented during the 2002-2003 school year. By no later than May 
1 , 2003, States must submit to the Department final information for all sections of the 
Consolidated State Application Accountability Workbook. 

Transmittal Instructions 

To expedite the receipt of this Consolidated State Application Accountability Workbook, 
please send your submission via the Internet as a .doc file, pdf file, rtf or .txt file or 
provide the URL for the site where your submission is posted on the Internet. Send 
electronic submissions to conapp@ed.gov. 

A State that submits only a paper submission should mail the submission by express 
courier to: 

Celia Sims 

U.S. Department of Education 
400 Maryland Ave., SW 
Room 3W300 

Washington, D.C. 20202-6400 
(202) 401-0113 
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PART I: Summary of Required Elements for State Accountability 
Systems 

Instructions 

The following chart is an overview of States' implementation of the critical elements 
required for approval of their State accountability systems. States must provide detailed 
implementation information for each of these elements in Part II of this Consolidated 
State Application Accountability Workbook. 

For each of the elements listed in the following chart, States should indicate the current 
implementation status in their State using the following legend: 

F: State has a final policy, approved by all the required entities in the State (e.g., 

State Board of Education, State Legislature), for implementing this element in its 
accountability system. 

P: State has a proposed policy for implementing this element in its accountability 

system, but must still receive approval by required entities in the State (e.g., 
State Board of Education, State Legislature). 

W: State is still working on formulating a policy to implement this element in its 

accountability system. 
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Summary of Implementation Status for Required Elements of 
State Accountability Systems 



Status 



Principle 1 : All Schools 



State Accountability System Element 



P 1.1 Accountability system includes all schools and districts in the state. 

p 1 .2 Accountability system holds all schools to the same criteria. 

p 1 .3 Accountability system incorporates the academic achievement standards. 

p 1 .4 Accountability system provides information in a timely manner. 

p 1.5 Accountability system includes report cards. 

1 .6 Accountability system includes rewards and sanctions. 



Pr nciple 2: All Students 

P 2.1 The accountability system includes all students 

p 2.2 The accountability system has a consistent definition of full academic year. 

2.3 The accountability system properly includes mobile students. 



Principle 3: Method of AYP Determinations 



p 3.1 Accountability system expects all student subgroups, public schools, and LEAs to reach 
proficiency by 2013-14. 

3.2 Accountability system has a method for determining whether student subgroups, public 
P schools, and LEAs made adequate yearly progress. 

p 3.2a Accountability system establishes a starting point. 

p 3.2b Accountability system establishes statewide annual measurable objectives. 

p 3.2c Accountability system establishes intermediate goals. 

Principle 4: Annual Decisions 

P 4.1 The accountability system determines annually the progress of schools and districts. 



STATUS Legend: 

F - Final state policy 

P - Proposed policy, awaiting State approval 
W - Working to formulate policy 
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Principle 5: Subgroup Accountability 

5.1 The accountability system includes all the required student subgroups. 

5.2 The accountability system holds schools and LEAs accountable for the progress of student 

subgroups. 

5.3 The accountability system includes students with disabilities. 

5.4 The accountability system includes limited English proficient students. 

5.5 The State has determined the minimum number of students sufficient to yield statistically 

reliable information for each purpose for which disaggregated data are used. 

5.6 The State has strategies to protect the privacy of individual students in reporting 
achievement results and in determining whether schools and LEAs are making adequate 
yearly progress on the basis of disaggregated subgroups. 

Pr nciple6: Based on Academic Assessments 

P 6.1 Accountability system is based primarily on academic assessments. 

Pr nciple7: Additional Indicators 

P 7.1 Accountability system includes graduation rate for high schools. 

7.2 Accountability system includes an additional academic indicator for elementary and middle 
P schools. 

p 7.3 Additional indicators are valid and reliable. 

Principle 8: Separate Decisions for Readinq/Lanquaqe Arts and Mathematics 




p 8.1 Accountability system holds students, schools and districts separately accountable for 
reading/language arts and mathematics. 



Principle 9: Svstem Validitv and Reliabilit 



P 9.1 Accountability system produces reliable decisions. 

p 9.2 Accountability system produces valid decisions. 

p 9.3 State has a plan for addressing changes in assessment and student population. 

Pr nciple 10: Participation Rate 

p 10.1 Accountability system has a means for calculating the rate of participation in the statewide 

assessment. 

p 1 0.2 Accountability system has a means for applying the 95% assessment criteria to student 
subgroups and small schools. 

STATUS Legend: 

F - Final policy 

P - Proposed Policy, awaiting State approval 
W- Working to formulate policy 
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PART II: State Response and Activities for Meeting State 
Accountability System Requirements 



Instructions 

In Part II of this Workbook, States are to provide detailed information for each of the 
critical elements required for State accountability systems. States should answer the 
questions asked about each of the critical elements in the State's accountability system. 
States that do not have final approval for any of these elements or that have not 
finalized a decision on these elements by January 31 , 2003, should, when completing 
this section of the Workbook, indicate the status of each element that is not yet official 
State policy and provide the anticipated date by which the proposed policy will become 
effective. In each of these cases, States must include a timeline of steps to complete to 
ensure that such elements are in place by May 1 , 2003, and implemented during the 
2002-2003 school year. By no later than May 1 , 2003, States must submit to the 
Department final information for all sections of the Consolidated State Application 
Accountability Workbook. 
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PRINCIPLE 1. A single statewide Accountability System applied to all public 
schools and LEAs. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING STATUTORY 
REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


1.1 How does the State 

Accountability System include 
every public school and LEA in 
the State? 


Every public school and LEA is 
required to make adequate 
yearly progress and is included 
in the State Accountability 
System. 

State has a definition of “public 
school” and “LEA” for AYP 
accountability purposes. 

• The State Accountability 
System produces AYP 
decisions for all public 
schools, including public 
schools with variant grade 
configurations (e.g., K-12), 
public schools that serve 
special populations (e.g., 
alternative public schools, 
juvenile institutions, state 
public schools for the 
blind) and public charter 
schools. It also holds 
accountable public 
schools with no grades 
assessed (e.g., K-2). 


A public school or LEA is not 
required to make adequate 
yearly progress and is not 
included in the State 
Accountability System. 

State policy systematically 
excludes certain public schools 
and/or LEAs. 


STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 


1.1 The State will move toward enhancing the current accreditation system by adding a separate 
Adequate Yearly Progress (AYP) achievement-based system that is congruent with the 
requirements of the No Child Left Behind (NCLB) Act of 2001. The Montana State 
Accountability System will include every public school and every school district in the state. All 
public schools and districts will be required to make adequate yearly progress. The State 
Accountability System will produce AYP decisions for all public schools, including public 
schools with variant grade configurations (e.g., K-l, K-2, K-3, 5-6, etc.), and state operated 
schools that serve special populations (e.g., Montana State School for the Deaf and Blind, and 
the Department of Corrections’ Pine Hills School, etc.). There are no public charter schools in 
Montana. Public schools with no grades assessed will be included in the system by examining 
the schools into which they feed. The State Superintendent will incorporate AYP policies and 
procedures into the Office of Public Instruction (OPI) State and Federal Grants Handbook. 

The Montana definition for school is found at Montana Code Annotated (MCA) 20-6-501. The 
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trustees of a district shall designate the grade assignments for the schools of the district, but for 
the purposes of this title each school is known as: (1) an elementary school when it comprises 
the work of any combination of kindergarten, other preschool programs, or the first eight grades 
or their equivalents; a middle school is a school comprising the work of grades 4 through 8 or 
any combination of grades 4 through 8 that has been accredited as a middle school under the 
provisions of MCA 20-7-102. When an accredited junior high school is operated by the district, 
grades 7 and 8 or their equivalents may not be considered as elementary grades; (2) a high 
school when it comprises the work of one or more grades of schoolwork of the equivalents 
intermediate between the elementary schools and the institutions of higher education in the state 
of Montana. Types of high schools are designated as follows: (a) a junior high school is a 
school comprising the work of grades 7 through 9 or their equivalents that has been accredited as 
a junior high school under the provisions of MCA 20-7-102; (b) a senior high school is a school 
that comprises the work of grades 10 through 12 or their equivalents and that is operated in 
conjunction with a junior high school; and (c) a 4-year high school is a school comprising the 
work of grades 9 through 12 or their equivalents. 

Students who attend alternative education programs are included in their “parent” school (the 
public school they would have attended in the absence of the alternative program) for assessment 
and accountability purposes. This does not include the state operated schools since testing and 
accounting for scores occurs within those schools, not in schools the students would otherwise 
attend in their home districts. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING STATUTORY 
REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


1.2 How are all public schools 
and LEAs held to the same 
criteria when making an AYP 
determination? 


All public schools and LEAs are 
systematically judged on the 
basis of the same criteria when 
making an AYP determination. 

If applicable, the AYP definition is 
integrated into the State 
Accountability System. 


Some public schools and LEAs 
are systematically judged on the 
basis of alternate criteria when 
making an AYP determination. 


STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 


1.2 As a Title I compliance agreement state, Montana will use its current assessment, Montana 
Comprehensive Assessment System (MontCAS), Phase 1, consisting of the norm-referenced 
Iowa Tests at grades 4, 8, and 11 as the basis for adequate yearly progress for all public schools 
and districts in reading and math (determined separately) during the transition period of 2002-03 
until the state’s new assessment (MontCAS, Phase 2) and accountability system becomes 
effective with the first full administration of the new assessment in April 2004, with delayed 
identifications for that year as described below. 
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2002-03 Process - Use of Previous Method 

The Montana Office of Public Instruction will use its current assessment system and the former 
“Title I only” process to identify all public schools and districts in need of improvement. In 
terms of accountability, during the transition school year of 2002-03, each school and district 
must achieve an average score for all students in the school or district as a whole and for each 
disaggregated subgroup in the grade or grades tested in that school or district of 45 NCE or 
higher on the basis of combined data for up to three years (2000-01, 2001-02, and 2002-03) or 
the 2002-03 school year alone, dependent upon obtaining the minimum “n” of 10 students. The 
minimum “n” of 10 used for reporting will also be used for AYP determinations at both the 
school and district levels for this transition year except with regard to participation rate as noted 
below. All schools and districts with an average score of less than 45 NCE in any subgroup 
including all students will be deemed as not making adequate progress. The following chart 
shows how the performance levels have been designated and applied in the past for designating 
percentages in each performance level and for identifying Title I schools and districts for 
improvement: 



Chart 1 

Montana Performance Categories Used With Norm Referenced Testing 





Percentile Ranking 


Normal Curve Equivalent 


Stanine 




Low 


High 


Low 


High 


Low 


High 


Novice 


1 


23 


1.0 


34.7 


1 


3 


Nearing Proficiency 


24 


40 


34.8 


44.9 


4 


4 


Proficient 


41 


89 


45.0 


76.3 


5 


7 


Advanced 


90 


99 


46.4 


99.0 


8 


9 



In addition to the average NCE threshold of 45 NCE for each subgroup (including all students) 
schools and districts will be held to the following: 

• 95% participation rate requirement - The 95% participation rate will be calculated by 
comparing the actual number of students tested to the actual enrollment at time of testing 
multiplied by .95 OR actual enrollment minus two in those schools or districts with 
enrollment of less than 40 students. 

• The other required indicator of attendance rates for elementary and middle schools will 
be applied. The State Superintendent will establish the attendance rate standard by July 
1, 2003. Schools and districts will be considered as having met AYP if they met the 
standard or if they made improvement toward the standard. Note: This standard has 
now been set at 80 percent. 

• The other required indicator for high schools, graduation rate, will be applied by using 
the last year of complete data which are school year 2001-02 rates. Those high schools 
not meeting the State target will be judged on progress from the previous year toward 
meeting the State target. This target threshold has now been set at 80 percent. 
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2003-04 Process 

The compliance agreement specifies that Montana will set cut scores and academic achievement 
standards by July 31, 2004. The agreement further specifies that the state will review the 
standards setting work and formally adopt the cut scores and academic achievement standards by 
August 31, 2004. It is not possible to set cut scores earlier using pilot test data from April 2003 
because only the new custom-developed items were included, not the full test. Scores from the 
first full administration of the new assessment will be produced in September 2004 and used to 
set baseline data. 

In October 2004, the percent proficient and above for all subgroups in each school and district on 
the April 2004 assessment will be compared to the starting points in order to determine adequate 
yearly progress separately for reading and mathematics for each school and district. Those with 
percentages below the starting point as well as those not making the 95% participation rate or 
progress on the other indicator will be notified during late October/early November 2004 that 
adequate yearly progress was not made. 

Since these determinations using April 2004 test results cannot be made for districts with 
identified Title I schools early enough to notify parents of their school choice options by the first 
day of school, public school choice and supplemental services will be offered to parents and 
students as soon as possible in the 2004-05 school year or not later than the start of the second 
semester. In Montana, public school choice within the district is only possible in the seven 
largest districts plus a handful of medium sized districts with more than one school at a particular 
grade span. So for the vast majority of Montana’s 866 public schools and 444 districts, school 
choice will not be available anyway unless an interagency agreement with a neighboring district 
is practicable. Where public school choice is not possible, supplemental educational services 
will be offered in Title I schools in the first year of improvement if approved providers are 
available. When schools and districts are identified in July 2005 on the basis of the April 2005 
assessments, they will be notified by the first week in August so those who must do so will be 
able to notify parents and students of their school choice and supplemental services options prior 
to the first day of school. 

All Montana public schools and LEAs will be systematically judged on the basis of the same 
criteria when making an AYP determination. The accountability of public schools without grades 
assessed (i.e., K-2 schools) will be based on the aggregate score for all students tested in the 
school that the feeder school students currently attend. Students who attend alternative education 
programs are included in their “parent” school (the public school they would have attended in the 
absence of the alternative program) for assessment and accountability purposes. 

2005-06 Process 

Based on the need to complete a standards-setting process and develop new assessments to be 
administered for grades 3, 5, 6, and 7 beginning spring of 2006, we will need to make final AYP 
decisions following the start of the 2006-07 school year. Pending final AYP determinations, we 
will hold all schools and districts in their current AYP status and continue to apply NCLB 
consequences, as applicable. Included in these actions are the following: 

1. Any school previously identified for improvement will continue to provide that level of 
intervention at the beginning of the year. 
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2. Any school that is identified for improvement, or that may be identified for improvement 
based upon the 2005-2006 results, will begin preparing to offer the next level of 
interventions. 

3. Schools will act immediately on the preliminary data when it is released in November. 

4. Schools that begin offering interventions at the start of the school year will continue to 
offer those interventions for the remainder of the school year, even if the school is no 
longer identified for improvement. 

As part of this process, we will be establishing annual measurable objectives to reach 100% 
proficiency by 2013-14. The AYP determinations will be made in accordance with the following 
timeline. Montana will observe all timelines for reporting AYP decisions as listed in Attachment 
B of the USED letter of March 7, 2006, from Dr. Henry Johnson. 



June and July, 2006 
October 2006 

November 2006 

December 2006 



Conduct standards setting (cut scores) for grades 3-8 and 10 

Test scores for grades 3-8 and 10 scheduled to be received from 
contractor 

Set new starting points for AYP and issue AYP decisions, subject 
to appeal 

30 day review period for schools and districts to appeal, with all 
post-appeal AYP decisions made before start of second semester 



New starting points, AMO's, and intermediate goals are shown in the below chart: 
Montana Annual Measurable Objective Trajectory 




Reading 



Math 
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All students with disabilities in Montana public schools as defined under section 602(3) of the 
Individuals with Disabilities Education Act (IDEA) amendments of 1997 and limited English 
proficiency (LEP) participate in the Montana Comprehensive Assessment Program. The 
Individualized Education Program (IEP) Team determines how students with disabilities will 
participate in the statewide assessment program. A team of educators familiar with the language 
abilities of LEP students determines how LEP students will participate. These requirements are 
contained in ARM 10.56.101, the assessment rules of the Board of Public Education found in the 
Administrative Rules of Montana (ARM). The Montana CRT Alternate Assessment Program 
(based on alternate achievement standards ) yields reading and mathematics assessment results 
for students with the most significant cognitive disabilities for inclusion in AYP determinations, 
utilizing Montana’s approved exception to the 1% rule. 

Guidance will be provided for use by LEP teams in determining whether an LEP student should 
be included in the Alternate Assessment Program. — Most LEP students will be included in the 
regular assessment with accommodations if necessary. This additional option is intended to be 
limited to the very few newly arrived immigrant students and will provide criteria to be used in 
making judgments about a student’s abilities and the need to utilize the Alternate Assessment 
Program in extreme cases. 

Montana has identified four performance levels for the new assessments (MontCAS, Phase 2). 
See response 1.3. MontCAS, Phase 2 is comprised of partially custom-developed criterion- 
referenced assessments that include multiple measures in the areas of reading, mathematics and 
science. The assessments will be administered in grades 3 through 8 and grade 10 on the 
following timetable: 

April 2004 - Reading and Mathematics, Grades 4, 8, and 10 

April 2005 - Reading and Mathematics, Grades 4, 8, and 10 

April 2006 - Reading and Mathematics, Grades 3, 4, 5, 6, 7, 8, and 10 

April 2007 - Reading and Mathematics, Grades 3, 4, 5, 6, 7, 8, and 10 

April 2008 - Reading and Mathematics, Grades 3, 4, 5, 6, 7, 8, and 10 and Science, Grades 4, 8, 
and 10 

With MontCAS, Phase 2 data of April 2004, student scores from the Montana CRT Alternate 
Assessment Program will be aggregated with those from the MontCAS, Phase 2 for all students 
and each subgroup in reading and mathematics. Montana’s approved exception to the 1% rule 
will be used. All of the required subgroups, including students with disabilities and LEP 
students, who are enrolled in a public school for a full academic year will be included in the 
performance measures that determine AYP for reading and mathematics separately (except as 
noted in Section 5.4 with regard to LEP students in their first year in a United States school). 





EXAMPLES FOR 


EXAMPLES OF 


CRITICAL ELEMENT 


MEETING STATUTORY 


NOT MEETING 




REQUIREMENTS 


REQUIREMENTS 
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1 .3 Does the State have, at a 
minimum, a definition of 
basic, proficient and 
advanced student 
achievement levels in 
reading/language arts and 
mathematics? 


State has defined three levels of 
student achievement: basic, 
proficient and advanced 3 

Student achievement levels of 
proficient and advanced 
determine how well students are 
mastering the materials in the 
State’s academic content 
standards; and the basic level of 
achievement provides complete 
information about the progress of 
lower-achieving students toward 
mastering the proficient and 
advanced levels. 


Standards do not meet the 
legislated requirements. 


STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



1 System of State achievement standards will be reviewed by the Standards and Assessments Peer 
Review. The Accountability Peer Review will determine that achievement levels are used in determining 
AYP. 
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1.3 Administrative Rules of Montana (ARM) 10.54.2502 (3) provide the following definitions of 
performance levels for Montana Content and Performance Standards in five subjects including 
reading and mathematics: 

a) “Performance level” means the level of achievement in broad, general terms; 

b) “Advanced level” means superior performance; 

c) “Proficient level” means solid academic performance for each benchmark, reaching 
levels of demonstrated competency over challenging subject matter, including 
subject-matter knowledge, application of such knowledge to real-world situations, 
and analytical skills appropriate to the subject matter; 

d) “Nearing proficiency level” means a student has partial mastery of the prerequisite 
knowledge and skills fundamental for proficient work at each benchmark; and 

e) “Novice level” means a student is beginning to attain the prerequisite knowledge and 
skills that are fundamental for work at each benchmark. 

For each of the content standards in reading and mathematics, four levels of performance 
descriptors have been developed. All of the MontCAS, Phase 2 assessments will be aligned to 
the content standards and descriptors. 

The performance levels are applied to the current assessment in terms of standardized test scores 
and will also apply to tests under development and scheduled for implementation starting in 
2004, as specified in the compliance agreement between Montana and the U.S. Department of 
Education by establishing cut scores and academic achievement standards using Bookmark 
methodology in July 2004 after the first full administration of reading and math assessments in 
grades 4, 8, and 10. The same method will be used in July 2006 after the first full administration 
of reading and math assessments in grades 3, 5, 6, and 7. The same method will be used in July 
2008 after first full administration of science assessments in grades 4, 8, and 10. 
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CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING STATUTORY 
REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


1 .4 How does the State provide 
accountability and adequate 
yearly progress decisions 
and information in a timely 
manner? 


State provides decisions about 
adequate yearly progress in time 
for LEAs to implement the 
required provisions before the 
beginning of the next academic 
year. 

State allows enough time to 
notify parents about public school 
choice or supplemental 
educational service options, time 
for parents to make an informed 
decision, and time to implement 
public school choice and 
supplemental educational 
services. 


Timeline does not provide 
sufficient time for LEAs to fulfill 
their responsibilities before the 
beginning of the next academic 
year. 



STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



1.4 As a compliance agreement state, Montana will continue using the current system with 
2002-03 data to identify schools not making one or more years of progress with identification 
for improvement designation after two years of not making progress in the same subject area. 
Norm-referenced tests (the ITBS in grades 4 and 8 and the ITED in grade 1 1) were given in 
middle to late March 2003 with results returned to LEAs and schools prior to the end of the 
school year. District and school personnel begin to examine their aggregate data provided by 
the test publisher at this point. In addition, the state provides statewide reports under the 
following timeline: 

Mid-July Statewide aggregated results 

Mid to late July Statewide disaggregated results by school and district 

Late July Notification of AYP status to schools and districts and beginning 

of 30 day period for Appeals Process (See Chart 2, bottom row). 



New Assessments; MontCAS, Phase 2 

Results for tests currently under development with implementation beginning in April 2004 are 
scheduled for return to LEAs and schools prior to the end of the school year; however, in the 
first years of implementation of each new set of grades to be tested (2004 for grades 4, 8, and 
10; 2006 for grades 3, 5, 6, and 7), the results will be delayed until after standard setting in the 
summers as per Montana’s compliance agreement with the U.S. Department of Education. 

Montana will provide decisions about adequate yearly progress in time for LEAs to implement 
the required provisions of No Child Left Behind before the beginning of the next academic year 
except as noted above for school year 2004-05. 
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For the purpose of determining AYP, the Montana Office of Public Instruction will ensure that 
results of the new State academic assessments will be available to the LEAs in a timely manner. 
(See chart 2 below.) 

Chart 2. Timeline for Use with New Assessments (MontCAS, Phase 2) 


Timeline 


Activity 


Mid April February -March Test 
Administration Window (annually) 


Statewide CRT Assessment Administration 
Window ( MontCAS Phase 2) 


Anytime during Test Administration 
Window, Make-Ups will be done (annually) 


Statewide Assessment Make-Up Window 


At the beginning of the testing window 
(annually) 


Collection of information on students enrolled 
on this date and for full academic year 


Six to Eight Weeks from Assessment 
Administration 


Assessment vendor required to provide 
assessment results to the OPI 


July May (annually) 


Schools receive assessment results 


Late July (annually) 


Schools will be notified of preliminary AYP 
status 


No later than the first day of school 


LEA notification to parents regarding school 
choice and supplemental services 


No later than 30 days after preliminary 
proposed identification of Schools/LEAs not 
meeting AYP (annually) 


District/LEA Appeals Process Begins 

OPI renders final determination in response to 

appeal 





EXAMPLES FOR 


EXAMPLES OF 


CRITICAL ELEMENT 


MEETING STATUTORY 


NOT MEETING 




REQUIREMENTS 


REQUIREMENTS 


1 .5 Does the State 


The State Report Card includes all the 


The State Report Card does not 


Accountability System 


required data elements [see Appendix 


include all the required data 


produce an annual 


A for the list of required data 


elements. 


State Report Card? 


elements]. 


The State Report Card is not 




The State Report Card is available to 
the public at the beginning of the 
academic year. 


available to the public. 




The State Report Card is accessible in 
languages of major populations in the 
State, to the extent possible. 






Assessment results and other 
academic indicators (including 
graduation rates) are reported by 
student subgroups 




STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 
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1.5 The State Superintendent developed the first Montana Statewide Education Profile in April 
1999. Montana citizens participated in a series of town meetings with the State Superintendent 
to determine the indicators of quality they wanted included in the Montana Statewide Education 
Profile. The second edition of the statewide profile was developed in May 2001. The data 
included in the profile is posted on the Montana Office of Public Instruction Web site 
(www.opi.state.mt.us) . Student achievement on the Iowa Tests (MontCAS, Phase 1), in the 
aggregate and disaggregated for each of the required subgroups, is included on the Web site. 

Data elements required by the NCLB, as listed in Appendix A, not currently included on the 
Web site will be collected and reported. An NCLB Report Card for schools, LEAs, and the state 
will be published annually according to NCLB requirements for state reporting. The Montana 
Office of Public Instruction currently utilizes the Internet Reporting and Information Service 
(IRIS), a reporting tool for all schools, LEAs, and the public to access assessment results. 

As a compliance agreement state, Montana will complete a Web based School, District, and 
State Report Card using the current assessment data (MontCAS, Phase 1) as well as other 
required data by September 2003. The OPI has a specific plan, detailed requirements, and listed 
work products that have been released in an RFP that was due April 30, 2003. Work will begin 
in June 2003 and is expected to be complete by September 2003. See Attachment B for the Web 
Report Card RFP as originally released. Due to incomplete responses by the original due date, 
the RFP was re-released on April 21, 2003. (English is the language of the major populations in 
the state and district, but efforts will be made to make the report cards available in other 
languages present for which translation is possible and practicable.) 

The 2004 2003-04 Montana Report Card (which will be issued in October 2004 January 2005) 
will include information on student achievement at each proficiency level on the new state 
academic assessment (MontCAS, Phase 2) including the Montana CRT Alternate Assessment 
Program, disaggregated by (1) all students, (2) race/ethnicity, (3) gender, (4) disability, (5) 
migrant status, (6) limited English proficiency status, and (7) economically disadvantaged status. 
After the second year of MontCAS, Phase 2 test administration, the report card will include the 
most recent two-year results in student achievement in reading and mathematics performance 
levels. The percent of students not tested, graduation rates for secondary schools, and attendance 
rates for elementary /middle schools will be reported in aggregate and in disaggregated subgroups 
except in the case of graduation rates because it will be four years before disaggregated data will 
be fully available on graduates. 

The professional qualifications of teachers in the State and the number of such teachers teaching 
with emergency or provisional credentials will be provided on the State report card. The percent 
of classes not taught by highly qualified teachers will be disaggregated by high-poverty 
compared to low-poverty schools. 

The Montana Report Card will contain information on schools not making AYP according to 
NCLB, Section 1116. A listing of all schools that failed to make AYP for the year will be 
reported on both the district report card and the state report card. Each school report card will 
indicate whether that school made AYP or not. 

As specified in the Montana Title I Compliance Agreement, statewide assessment results from 
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the April 2004 administration of MontCAS, Phase 2 will be provided to the Montana Office of 
Public Instruction by the testing vendor in September 2004 and the NCLB Report Card will be 
made available to schools in October 2004 . In fall 2004, as soon as possible. The new 
assessment data will then be available for publication in the Report Card and Montana will 
become fully compliant with the NCLB legislation. 

While the Montana Office of Public Instruction is operating under the Title I Compliance 
Agreement, the OPI will report by school, district, and state the following information on the 
September 2003 report cards: 

• The Iowa Tests - Total Reading and Total Math Scores 

The percent of students scoring in each performance level for each subgroup for the 

school (See Chart 1 in this document) and aggregate NCE score for school 

The percent of students scoring in each performance level for each subgroup for the 

district (See Chart 1 in this document) and aggregate NCE score for district 

The percent of students scoring in each performance level for each subgroup for the state 

(See Chart 1 in this document) and aggregate NCE score for state 

• AYP Determination 

A listing of all schools that failed to make AYP for the year will be reported on both the 
district report card and the state report card; each school report card will indicate whether that 
school made AYP or not. 

Each school and district will be evaluated to see if the 45 NCE standard for assessment in 
2003 has been met for all subgroups including all students. There will be an indicator on 
the school report card as to whether the school and the district made AYP. 

• Participation Rate 

Participation rate for the state, each school, and district and percentage not tested will be 
displayed disaggregated for all subgroups including all students. 

• Additional Indicators 

The graduation rate for high schools or the attendance rate for elementary /middle schools 
will be displayed in the aggregate on the September 2003 Report Card as disaggregated 
data on these elements is not yet available 

• Teacher Quality (This will be added to Report Cards posted in September 2003 by early 
November 2003 in order to display current school year information. It will be added each 
subsequent year in early November for this same reason.) 

The professional qualifications of teachers, the percentage of teachers teaching with 
emergency or provisional credentials, and the percentage of classes taught by teachers who are 
not highly qualified in that school, district, and the state, in the aggregate and disaggregated by 
high-poverty compared to low-poverty schools 

The first production of the new revised Report Card will be in October 2004, after the 
administration of the new statewide assessments, standards setting, and receipt of scores, and 
final AYP determinations . This publication will include all that is required by NCLB. The 
trend data for the new assessments will be included as Montana administers the test in future 
years. 
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CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING STATUTORY 
REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


1 .6 How does the State 


State uses one or more types of 


State does not implement 


Accountability System 


rewards and sanctions, where the 


rewards or sanctions for public 


include rewards and 


criteria are: 


schools and LEAs based on 


sanctions for public 
schools and LEAs? 


• Set by the State; 

• Based on adequate yearly 
progress decisions; and, 

• Applied uniformly across 
public schools and LEAs. 


adequate yearly progress. 



STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



Although the state superintendent’s policy is not complete, she will review all schools and 
districts in the state following the definition for Adequate Yearly Progress (AYP) as contained in 
this document. In other words, the required criteria for AYP as contained in the No Child Left 
Behind Act will be followed with all school and districts. The Superintendent will provide 
rewards to those school districts and schools that meet or exceed the AYP requirements for two 
consecutive years. These rewards may include, but are not limited to, the following: 

1. Public recognition, including recognition at Montana Board of Public Education 
meetings, on the Montana Office of Public Instruction (OPI) website, and through the 
news media of the state. 

2. Preference for appointments to serve on task forces and advisory committees that 
advise the state superintendent; and 

3. Designation by the State Superintendent as a “School (and/or District) of Progress” 

In addition, the State Superintendent will provide sanctions to those school districts and schools 
that do not meet the AYP requirements for two consecutive years. The sanctions may include, 
but are not limited to, the following: 

1. Provision of a plan of improvement to be included in the Five-Year Comprehensive 
Education Plan required of all schools and districts by the Montana Accreditation 
Standards; 

2. Regular reporting of information concerning implementation of the Five-Year 
Comprehensive Education Plan. 

3. Notification, through the news media and OPI website, concerning the schools’ and 
districts’ failure to meet AYP; and 

4. School personnel will be required to participate in intensive and sustained 
professional development. 

5. In addition to the above sanctions, all Title I districts and schools will be subject to 
the sanctions established in NCLB Section 1116. 
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PRINCIPLE 2. All students are included in the State Accountability System. 





EXAMPLES FOR 


EXAMPLES OF 


CRITICAL ELEMENT 


MEETING STATUTORY 


NOT MEETING 




REQUIREMENTS 


REQUIREMENTS 


2.1 How does the State 


All students in the State are included in 


Public school students exist in 


Accountability System 


the State Accountability System. 


the State for whom the State 


include all students in 




Accountability System makes no 


the State? 


The definitions of “public school” and 
“LEA” account for all students enrolled 
in the public school district, regardless 
of program or type of public school. 


provision. 



STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



2.1 Administrative Rules of Montana (ARM) 10.56.101(7) require that all students in the public 
schools shall participate in the state level assessments. This accounts for all students enrolled in 
public schools regardless of program or type of public school. The same definition from MCA 
20-6-501 for public schools cited in Sec. 1.1 applies here. 

For the current statewide assessment, The Montana Guide for Test Coordinators and 
Administrators with 2003 Updates http://www.opi. state. mt.us/PDF/Assessment/02Guide&03Update.pdf lists 
options for participation, including alternate assessment on page 1. In addition, the Assessment 
Handbook, Volume 3 includes a section, “Options and Accommodations to Support All Students 
in the StatewideAssessment” http://www.opi.state.mt.us/pdf/Assessment/2002handbookV3.pdf 
Similar options will be developed for tests currently in development and scheduled to begin 
April 2004. 

A similar policy will be developed for inclusion in the OPI State and Federal Grants Handbook 
to include all students in the accountability system. Public schools with no grades assessed will 
be included in the system by examining the schools into which they feed. Students who attend 
alternative education programs are included in their “parent” school (the public school they 
would have attended in the absence of the alternative program) for assessment and accountability 
purposes. 

All students with disabilities in Montana public schools as defined under section 602(3) of the 
Individuals with Disabilities Education Act (IDEA) amendments of 1997 and limited English 
proficiency (LEP) participate in the Montana Comprehensive Assessment Program. The 
Individualized Education Program (IEP) Team determines how students with disabilities will 
participate in the statewide assessment program. A team of educators familiar with the language 
abilities of LEP students determines how LEP students will participate. These requirements are 
contained in ARM 10.56.101, the assessment rules of the Board of Public Education found in the 
Administrative Rules of Montana (ARM). The Montana CRT Alternate Assessment (based on 
alternate achievement standards ) yields Reading and Mathematics assessment results for 
students with significant cognitive disabilities for inclusion in AYP determinations, utilizing 
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Montana’s approved exception to the 1% rule. 

All of the required subgroups, including students with disabilities and with limited English 
proficiency, who are enrolled in a school for a full academic year will be included in the 
performance level measures that determine AYP for the school. The same will apply for all 
those enrolled in the district for a full academic year in measures that determine AYP for the 
district. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING STATUTORY 
REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


2.2 How does the State define 
“full academic year” for 
identifying students in AYP 
decisions? 


The State has a definition of “full 
academic year” for determining 
which students are to be included 
in decisions about AYP. 

The definition of full academic 
year is consistent and applied 
statewide. 


LEAs have varying definitions of 
“full academic year.” 

The State’s definition excludes 
students who must transfer from 
one district to another as they 
advance to the next grade. 

The definition of full academic 
year is not applied consistently. 


STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 


2.3 Full academic year is defined as continuous enrollment from the October enrollment 
reporting date (first Monday in October) through the March test administration (current 
assessment). Continuous enrollment means that a student is enrolled in the school unless he 
or she has withdrawn, been expelled, or dropped out. 

The same definition will apply for tests currently in development except it will be the “April test 
administration.” This definition is applied separately at both the school and district level when 
determining if adequate yearly progress has been made at the school level and at the district 
level. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING STATUTORY 
REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


2.4 How does the State 
Accountability System 
determine which students 
have attended the same 


State holds public schools 
accountable for students who 
were enrolled at the same public 
school for a full academic year. 


State definition requires students 
to attend the same public school 
for more than a full academic 
year to be included in public 
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public school and/or LEA for 
a full academic year? 



State holds LEAs accountable for 
students who transfer during the 
full academic year from one 
public school within the district to 
another public school within the 
district. 



school accountability. 

State definition requires students 
to attend school in the same 
district for more than a full 
academic year to be included in 
district accountability. 



State holds public schools 
accountable for students who 
have not attended the same 
public school for a full academic 
year. 



STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



2.3 Montana began collecting information on full-year/partial-year attendance (see 2.2) on the 
norm-referenced tests’ (ITBS and ITED) answer documents in 2002. Guidelines for coding 
that information are available in The Montana Guide for Test Coordinators and 
Administrators with 2003 Updates http://www.opi. state. mt.us/PDF/Assessment/02Guide&03Update.pdf . 
Similar procedures will be employed for tests currently in development and scheduled to 
begin April 2004. 

For inclusion in AYP determination on the current assessment (MontCAS. Phase 1: 

All student subgroups, including all the students group, will be held accountable to the AYP 
indicators: 

A student who is enrolled continuously in that school from the first Monday in October 
through the March testing administration will be included when determining if the school 
has made adequate yearly progress. 

A student who is enrolled continuously in the district from the first Monday in October 
through the March testing administration will be included when determining if the school 
has made adequate yearly progress. 

For inclusion in AYP determination on the new assessments (MontCAS, Phase 2: 

All student subgroups, including all the students group, will be held accountable to the AYP 
indicators: 

A student who is enrolled continuously in that school from the first Monday in October 
through the April testing administration will be included when determining if the school 
has made adequate yearly progress. 

A student who is enrolled continuously in the district from the first Monday in October 
through the April testing administration will be included when determining if the school 
has made adequate yearly progress. 
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Montana does not have a system for determining if a student has been continuously enrolled in 
the state . As a result, all students tested will be included when determining if the state has made 
adequate yearly progress. 



PRINCIPLE 3. State definition of AYP is based on expectations for growth in 
student achievement that is continuous and substantial, such that all students 
are proficient in reading/language arts and mathematics no later than 2013-2014. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


3.1 How does the State’s 

definition of adequate yearly 
progress require all students 
to be proficient in 
reading/language arts and 
mathematics by the 201 3- 
2014 academic year? 


The State has a timeline for 
ensuring that all students will 
meet or exceed the State’s 
proficient level of academic 
achievement in reading/language 
arts 2 and mathematics, not later 
than 2013-2014. 


State definition does not require 
all students to achieve 
proficiency by 2013-2014. 

State extends the timeline past 
the 201 3-2014 academic year. 


STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 
In accordance with the requirements of NCLB 



2 If the state has separate assessments to cover its language arts standards (e.g., reading and writing), 
the State must create a method to include scores from all the relevant assessments. 
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3.1 As a compliance agreement state, Montana will establish a timeline for determining whether 
ah students meet or exceed the proficient level of academic achievement in reading and 
mathematics, not later than the 2013-14 school year based on new assessments first administered 
in April 2004. See response to question 1.2 with regard to establishing the starting points, annual 
measurable objectives, and intermediate goals as a result of a new assessment being implemented 
in spring 2004 and additional grades added in spring 2006. The starting points and targets will 
be set separately for reading and math. 

Montana’s definition of adequate yearly progress (AYP) requires all students to be proficient in 
reading and mathematics by the end of the 2013-14 school year and requires ah students and 
each subgroup to be held accountable to meet ah of the academic indicators used to measure 
AYP (percent proficient in reading and mathematics; percent of participation in the assessments; 
graduation rate for secondary schools; and attendance rate for elementary and middle schools.) 
(See Chart 3 below.) 

The Montana definition of AYP will be included in the OPI State and Federal Grants Handbook. 



Chart 3. Accountability Subgroups and Academic Indicators 





j| Academic Indicators j 


| Participation Rate j] 


Graduation/Attendance 
Rate * 


Reading 

% 

Meeting 

Standard 


Mathematics 
% Meeting 
Standard 


Reading 


Mathematics 


All Students 












Economically 

Disadvantaged 












R/E White 










R/E Black 




















R/E Asian 










R/E Am. 
Indian/Alaskan 










Students with 
Disabilities 










LEP Students 











* The school/LEA will not be required to disaggregate graduation rate and attendance rate data 
into the subgroups for accountability unless the school /LEA is using the “Safe Harbor” 
provision to meet AYP. Until the disaggregations on these indicators are available at the state 
level, schools and districts will be asked to provide the data needed for “Safe Harbor” 



25 








CONSOLIDATED STATE APPLICATION ACCOUNTABILITY WORKBOOK 



determinations. 

All subgroups identified in Chart 3 will be held accountable to the academic indicators of 
reading and mathematics. Montana’s Compliance Agreement with the U.S. Department of 
Education specifies that a timeline be established for public schools to reach the goal of 100% of 
students proficient in reading and mathematics by the end of the 2013-14 school year. Annual 
and intermediate goals will be established, using the school year 2005-06 data from grades 3-8 
and 10 as soon as it is available, to assure increases in the percent of students proficient in 
reading and math over the next eight years. 

As Montana defines annual measurable objectives and intermediate goals, the first increase is 
expected in 2005-06, followed by intermediate incremental increases to assure that Montana 
public schools and districts meet the goal of 100% proficiency in 2013-14. 

The development of intermediate goals includes the following assumptions for Montana: 

1) Calculate the starting point for determining AYP based on 2003-04 assessment data that 
follows the recommendations of the U.S. Department of Education. This percent will 
serve as the standard for AYP determinations for the 2004-05 school year. 

2) Based on the starting points, establish first intermediate goal and incremental increase. 

3) Recalculate the starting point, using the 2005-06 data from grades 3 through 8 and 10. 
These averages will be used to determine intermediate goals and annual measurable 
objectives by grade configuration for the next seven years. This trajectory will assure 
that all students will reach the proficient level of performance by 2013-14. 

Montana will use the process for calculating the initial starting points that is outlined in Chart 5 
in section 3.2a, using assessment results from the MontCAS, Phase 2 that will be administered in 
spring 2004 in grades 4, 8, and 10. The same method will be used to reestablish new starting 
points when assessment data from grades 3-8 and 10 are incorporated. 

GROWTH OBJECTIVE (“Safe Harbor” Provision) 

If any student sub-groups do not meet or exceed the state annual measurable objectives, the 
public school or LEA may be considered to have met AYP if the percent of students in the non- 
proficient subgroup: 

1) Decreased by 10% on the reading and mathematics indicators from the preceding school 
year, and 

2) Made progress on the other indicator, or is at/above the target goal for that indicator. 

3) Meets the participation rate. 



! I 


CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 
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CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


3.2 How does the State 
Accountability System 
determine whether each 
student subgroup, public 
school and LEA makes 
AYP? 


For a public school and LEA to 
make adequate yearly progress, 
each student subgroup must 
meet or exceed the State annual 
measurable objectives, each 
student subgroup must have at 
least a 95% participation rate in 
the statewide assessments, and 
the school must meet the State’s 
requirement for other academic 
indicators. 


State uses different method for 
calculating how public schools 
and LEAs make AYP. 




However, if in any particular year 
the student subgroup does not 
meet those annual measurable 
objectives, the public school or 
LEA may be considered to have 
made AYP, if the percentage of 
students in that group who did 
not meet or exceed the proficient 
level of academic achievement 
on the State assessments for that 






year decreased by 1 0% of that 
percentage from the preceding 
public school year; that group 
made progress on one or more of 
the State’s academic indicators; 
and that group had at least 95% 
participation rate on the 
statewide assessment. 




STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 
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EXAMPLES FOR 


EXAMPLES OF 


CRITICAL ELEMENT 


MEETING REQUIREMENTS 


NOT MEETING 






REQUIREMENTS 



3.2 With the new assessments in place, Montana’s State Accountability System will base 
its annual determination of whether each subgroup, public school and LEA makes 
AYP on the achievement of all students, including these subgroups: economically 
disadvantaged, racial/ethnic, students with disabilities, and limited English 
proficient. Categories for racial/ethnic are as follows: 



R/E White 
R/E Black 
R F Hispanic 
R/E Asian 
R/E Am. 
Indian/Alaskan 



Montana’s AYP calculation also incorporates the other academic indicators of graduation 
rate (for secondary schools) and attendance rate (for elementary and middle schools) and 
participation rate for all. (See Chart 3.) 

(NOTE: For accountability purposes, the school or LEA will not be required to disaggregate 
graduation rate and attendance rate data into the subgroups unless the school or LEA is using the 
“Safe Harbor” provision to meet AYP.) 

Montana will use a decreasing trend calculation under the “Safe Harbor” provision to identify 
schools that failed to meet AYP by the method outlined in Chart 4. A Montana public school or 
LEA may be considered to have made AYP if the percent of students in the non-proficient 
subgroup: 

1) Decreased by 10% from the preceding school year, and 

2) Made progress on the other academic indicator, or is at/above the target for that academic 
indicator, and 

3) Attained a 95% participation rate (to be calculated as explained previously) 
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EXAMPLES FOR 


EXAMPLES OF 


CRITICAL ELEMENT 


MEETING REQUIREMENTS 


NOT MEETING 






REQUIREMENTS 



Chart 4. “Safe Harbor” Provision for AYP Determination with Accountability Subgroups 
and Indicators 





Academic Indicators 


Participation Rate 


Graduation/Attendance 

Rate* 


Reading 
% Meeting 
Standard 


Mathematics 
% Meeting 
Standard 


Reading 


Mathematics 




Decrease 

by 10% 
that 

percent of 
non- 
proficient 
students 
from the 
preceding 
year in the 
school 


Decrease 

by 10% 
that 

percent of 
non- 
proficient 
students 
from the 
preceding 
year in the 
school 


Attained a 
95% 

Participation 

Rate 


Attained a 
95% 

Participation 

Rate 


Meets or shows 
progress toward this 
indicator by that sub- 
group 


All Students 










Economically 

Disadvantaged 












R/E White 










R/E Black 










R/E Hispanic 










R/E Asian 










R/E Am. 
Indian/Alaskan 










Students with 
Disabilities 










LEP Students 











*The school/LEA will not be required to disaggregate graduation rate and attendance rate data into the 
subgroups for accountability unless the school/LEA is using the “Safe Harbor” provision to meet AYP. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


3.2a What is the State’s starting 
point for calculating 
Adequate Yearly 
Progress? 


Using data from the 2001-2002 
school year, the State 
established separate starting 
points in reading/language arts 
and mathematics for measuring 
the percentage of students 


The State Accountability System 
uses a different method for 
calculating the starting point (or 
baseline data). 
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CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 




meeting or exceeding the State’s 
proficient level of academic 
achievement. 

Each starting point is based, at a 
minimum, on the higher of the 
following percentages of students 
at the proficient level: (1 ) the 
percentage in the State of 
proficient students in the lowest- 
achieving student subgroup; or, 
(2) the percentage of proficient 
students in a public school at the 
20 th percentile of the State’s total 
enrollment among all schools 
ranked by the percentage of 
students at the proficient level. 

A State may use these 
procedures to establish separate 
starting points by grade span; 
however, the starting point must 
be the same for all like schools 
(e.g., one same starting point for 
all elementary schools, one same 
starting point for all middle 
schools...). 
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STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



3.2a Starting with data from the 2003-04 school year on the new assessment, Montana will set 
separate starting points for reading and mathematics for public schools based on MontCAS, 
Phase 2 and Montana Alternate Assessment results. The starting points calculated based on 2004 
assessment data will serve as the AYP standard for 2003-04, 2004-05, and the basis for an 
increased intermediate goal set for 2005-06. As outlined in Sec. 3.1, starting points will be 
recalculated in the fall of 2006 to incorporate results from grades 3, 5, 6, and 7. The new starting 
points will be used to determine intermediate goals and annual measurable objectives for the 
remaining years to reach 100% proficiency by 2013-14. 

Chart 5. Calculating the Starting Point for AYP 

The calculation of each starting point will be based on the higher of the following percentages of 
students at or above proficient: (1) the percentage of students at or above proficient in the 
public school at the 20 th percentile of the State’s total enrollment among all schools ranked in 
ascending order by the percentage of students at or above proficient; or, (2) the statewide 
percentage of students at or above proficient in the lowest-achieving student subgroup. 

Method 1 

In summer fall 2004, rank all Montana public schools in order according to the percent of 
students who scored at the proficient level or above in reading. Montana may wish to calculate 
different starting points for public elementary, middle and high schools. The same process is 
used to calculate the starting point for mathematics. (In Steps 1 through 5, references are made 
to Chart 5a, Example A, found on the following page.) 

1. In a chart similar to Example A, record the total students in the enrollment records for 
each school after they have been ordered based on the percent of students who scored at 
the proficient level or above. 

2. Beginning with the school with the smallest percent of proficient students in reading, 
calculate the cumulative enrollment. Referring to Example A, the cumulative enrollment 
for School X is 397 {200 (School Z) + 65 (School Y) + 132 (School X)}. 

3. Multiply the total student enrollment for Montana public schools (top cumulative 
enrollment number) by 20 percent (.20) to find 20 percent of the total student enrollment. 
In the example, 20 percent of 1619 is 323.8. Rounding yields 324. 

4. Count up from the school with the smallest percent of students proficient in reading to 
identify the public schools whose combined school populations represent 20 percent of 
the total student enrollment (cumulative enrollment). From Example A, 20 percent of the 
total student enrollment is 324. To reach this number, the student populations from 

School X, School Y, and School Z are combined. 
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5. Use the percent of students who scored at the proficient level in reading and mathematics 
from the public schools identified in Step 4. This percent is the minimum starting point 
for reading and mathematics. In Chart 5a, Example A, the minimum starting point is 30 
percent (the percent of proficient students at School X). 

Method 2 

In summer fall 2004, calculate the statewide percentage of students at or above proficient in 
the lowest-achieving student subgroup for each subject tested. 

For each subject tested, use the greater of the percentages calculated using Method 1 and 
Method 2 as the starting point 



Chart 5a. Example A- Method 1 



School Name 


Percent of Students 
Proficient in 
Reading 


Total students in 
enrollment records 


Cumulative enrollment 


School A 


54% 


235 


1619 (1384 + 235) 


School B 


40% 


400 


1384 (984 + 400) 


School W 


38 % 


587 


984 (397 + 587) 


School X 


30% 


132 


397 (265 + 132) 


School Y 


29 % 


65 


265 (200 + 65) 


School Z 


20% 


200 


200 



Each intermediate goal will reflect annual measurable objectives based on state performance as 
defined in the federal legislation. Additionally, the school growth (“Safe Harbor” provision 
described in Section 3.1) calculation will be applied if the school or LEA did not meet AYP state 
annual objectives. The intermediate goals and annual objectives established by Montana will 
guide public schools in reaching the target goal of 100% proficiency by the end of the 2013-14 
school year. 

Note: Starting points have now been set using Method 1. The starting points are as follows: 
Reading (grades 4, 8, and 10) - 55% proficient or above 
Math (grades 4, 8, and 10) - 40% proficient or above 
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CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


3.2b What are the State’s annual 
measurable 

objectives for determining 
adequate yearly progress? 


State has annual measurable 
objectives that are consistent 
with a state’s intermediate goals 
and that identify for each year a 
minimum percentage of students 
who must meet or exceed the 
proficient level of academic 
achievement on the State’s 
academic assessments. 


The State Accountability System 
uses another method for 
calculating annual measurable 
objectives. 

The State Accountability System 
does not include annual 
measurable objectives. 




The State’s annual measurable 
objectives ensure that all 
students meet or exceed the 
State’s proficient level of 
academic achievement within the 
timeline. 






The State’s annual measurable 
objectives are the same 
throughout the State for each 
public school, each LEA, and 
each subgroup of students. 





STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



3.2b As Montana defines annual and intermediate goals, the first increase is expected in 2005- 
06, followed by incremental increases in the intermediate goals and annual goals that may 
remain the same for more than one year to assure that Montana public schools and districts meet 
the goal of 100% proficiency in 2013-14. These will identify a single percent of students who 
must annually meet or exceed the proficient level of performance on MontCAS, Phase 2 or the 
Montana Alternate Assessment. 

The development of intermediate goals and annual measurable objectives includes the following 
assumptions for Montana: 

1) Calculate the starting point for determining AYP based on 2003-04 assessment data that 
follows the recommendations of the United States Department of Education. (See Chart 3 
in this document.) This percent will serve as the standard for AYP determinations for the 
2004-05 school year. 

2) Based on the starting points and data from 2003-04 and 2004-05, establish first 
intermediate goal and incremental increase. 

3) Recalculate the starting point, using the 2005-06 data from grades 3-8 and 10. These 
averages will be used to determine intermediate goals and annual measurable objectives 
by grade configuration for the next seven years. This trajectory will assure that all 
students will reach the proficient level of performance by 2013-14. 



33 





CONSOLIDATED STATE APPLICATION ACCOUNTABILITY WORKBOOK 



Montana will use the process for calculating the initial starting points that is outlined in Chart 5 
in this document, using assessment results from the MontCAS, Phase 2 that will be administered 
in spring 2004 in grades 4, 8, and 10. The same method will be used to reestablish new starting 
points when assessment data from grades 3-8 and 10 are incorporated. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


3.2c What are the State’s 
intermediate goals for 
determining adequate 
yearly progress? 


State has established 
intermediate goals that increase 
in equal increments over the 
period covered by the State 
timeline. 

• The first incremental 

increase takes effect not 
later than the 2004-2005 
academic year. 


The State uses another method 
for calculating intermediate goals. 

The State does not include 
intermediate goals in its definition 
of adequate yearly progress. 




• Each following incremental 
increase occurs within 
three years. 




STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 


3.2c Same answer as 3.2b 
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PRINCIPLE 4. State makes annual decisions about the achievement of all public 
schools and LEAs. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


4.1 How does the State 
Accountability System 
make an annual 
determination of whether 
each public school and LEA 
in the State made AYP? 


AYP decisions for each public 
school and LEA are made 
annually. 3 


AYP decisions for public schools 
and LEAs are not made annually. 



STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



4.1 The State will make an annual determination of whether each public school and district in the 
State achieved AYP. 

Information used for AYP determination includes: 

• the proficiency status of each student tested in the state based on the assessment results for 
the student. (Each student will have a total mathematics and a total reading score and 
students’ proficiency will be determined for each test as provided by the testing company 
contracted to score and report test results.); 

• whether each student has completed a full academic year at the school, or LEA level as 
determined by a comparison of the roster of students enrolled in October who were 
continuously enrolled through the testing window for the spring assessments; 

• the number of students enrolled for a full academic year determined by comparing the 
number of continuously enrolled students with the number of tested students; 

• the percent of students enrolled for a full academic year; 

• the graduation rate for public high schools as determined by the formula indicated in Section 
7.1; 

• the attendance rate for public elementary and middle schools as determined through the ADA 
process (average daily attendance); and 

• disaggregated test results, percent tested, graduation rate, and attendance rate across the 
required subgroups. 

All required subgroups will be identified based on subgroup membership indicated in the spring 
enrollment collection. Montana will notify schools/LEAs of any subgroup that initially does not 
meet AYP in one year on any indicator (i.e., reading, mathematics, including participation rate 
for each , attendance rate, or graduation rate); however, if that school/LEA successfully meets 
AYP for that same indicator the following year, that school/LEA will be considered to have met 



3 Decisions may be based upon several years of data and data may be averaged across grades within a 
public school [§1111 (b)(2)(J)]. 
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the AYP standard and will not be identified for school improvement. 

Each school, LEA and sub-group will be required to meet the annual objectives and intermediate 
goals. Each school and LEA, including all subgroups, will be required to meet the 95% 
assessment participation rate indicator. 

Montana will establish a statewide standard for the graduation rate indicator (for secondary 
schools) and the attendance rate indicator (for elementary and middle schools). Montana will 
apply a growth standard to public schools that did not meet or exceed the standard for graduation 
or attendance. (See Principle 7.) These standards have been set at 80 percent or progress 
toward 80 percent in both indicators. 

Public schools will be accountable for all students who have been enrolled in the school for a full 
academic year. The LEA is accountable for all students who have been enrolled for a full 
academic year in that LEA. The SEA is accountable for all students who were tested in the 
spring assessments regardless of whether enrolled for a full academic year in the state. (See 
Section 2.2.) 

The decision about whether a school has made AYP is currently the responsibility of the Office 
of Public Instruction (OPI). All accountability decisions will be based on the information 
collected by the OPI through its various fiscal and non-fiscal data information systems: 

October and February Enrollment/ AD A through Montana Automated Education Finance 
and Information Reporting System (MAEFAIRS) 

Spring Enrollment of Students (Validation Form) 

Annual Data Collection (ADC) in October 
Assessment Results by Student 
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PRINCIPLE 5. All public schools and LEAs are held accountable for the 
achievement of individual subgroups. 



CRITICAL ELEMENT 



EXAMPLES FOR 
MEETING REQUIREMENTS 



EXAMPLES OF 

NOT MEETING REQUIREMENTS 



5.1 How does the definition of 
adequate yearly progress 
include all the required 
student subgroups? 



Identifies subgroups for defining 
adequate yearly progress: 
economically disadvantaged, 
major racial and ethnic groups, 
students with disabilities, and 
students with limited English 
proficiency. 

Provides definition and data 
source of subgroups for adequate 
yearly progress. 



State does not disaggregate data 
by each required student 
subgroup. 



STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



5.1 Montana’s definition of AYP includes measuring and reporting the achievement of 
subgroups of students by the indicators and subgroups that appear in Chart 6 (Accountability 
Subgroups and Academic Indicators). Currently, Montana reports school, LEA, and state 
performance by the required student subgroups on MontCAS, Phase 1 (The Iowa Tests). This 
report can be viewed at the Montana Office of Public Instruction Web site at 
http://www.opi.state.mt.us. 



Chart 6. Accountability Subgroups and Aca< 


iemic Indicators 




Academic Indicators 


Participation Rate 


Graduation/ Attendance 
Rate* 


Reading 

% 

Meeting 

Standard 


Mathematics 
% Meeting 
Standard 


Reading 


Mathematics 




All Students 












Economically 

Disadvantaged 












R/E White 










R/E Black 










R/E Hispanic 










R/E Asian 










R/E American 
Indian/ Alaskan 










Students with 
Disabilities 










LEP Students 










* The school/LEA will not be required to d 


isaggregate graduation rate and attendance rate data 
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into the subgroups for accountability unless the school/LEA is using the “Safe Harbor” provision 
to meet AYP. 

Montana’s definition of adequate yearly progress (AYP) requires all student subgroups to be 
proficient in reading and mathematics by the end of the 2013-14 school year. 

Montana has several data collection systems that maintain all student, school, LEA, and state 
data. This data is disaggregated and reported for all schools, LEAs, and the state currently using 
the IRIS system (for Iowa Test results used for 2002-03 determinations). 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 

NOT MEETING REQUIREMENTS 


5.2 How are public schools 
and LEAs held 
accountable for the 
progress of student 
subgroups in the 
determination of adequate 
yearly progress? 


Public schools and LEAs are held 
accountable for student subgroup 
achievement: economically 
disadvantaged, major ethnic and 
racial groups, students with 
disabilities, and limited English 
proficient students. 


State does not include student 
subgroups in its State 
Accountability System. 



STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



5.2 In determining whether each school and district, as well as the state as a whole, meets 
statewide annual measurable objectives, Montana will calculate, for each subgroup, the percent 
of the tested students who achieve the proficient level or higher, examine participation rates, 
examine progress on the other indicator (graduation for high schools/attendance rate for 
elementary and middle schools) and employ the safe harbor provision. The State may decide at a 
future time to employ uniform averaging procedures to obtain valid data for small sample sizes 
over time. Each subgroup within the school or LEA must meet the objective for each indicator 
in order to make AYP. 
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The participation rate is determined as having been achieved by either verifying that the number 
of students tested is equal to or greater than 95% of the actual enrollment or equal to or greater 
than the actual enrollment minus two for schools/districts with less than 40 students. 

Montana will notify schools/LEAs of any subgroup that initially does not meet AYP in one year 
on any indicator (i.e., reading, mathematics (i including participation rate in each), attendance 
rate, or graduation rate); however, if that school/LEA successfully meets AYP for that same 
indicator the following year, that school/LEA will be considered to have met the AYP standard 
and will not be identified for school improvement. 

The Montana Report Card will chart the progress of all groups of students and the status of each 
group in relation to annual measurable objectives based on the percent of students at the 
proficient level for reading and mathematics, the participation rate and the other academic 
indicators. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 

NOT MEETING REQUIREMENTS 


5.3 How are students with 
disabilities included in the 
State’s definition of 
adequate yearly progress? 


All students with disabilities 
participate in statewide 
assessments: general 
assessments with or without 
accommodations or an alternate 
assessment based on grade level 
standards for the grade in which 
students are enrolled. 

State demonstrates that students 
with disabilities are fully included 
in the State Accountability 
System. 


The State Accountability System 
or State policy excludes students 
with disabilities from participating 
in the statewide assessments. 

State cannot demonstrate that 
alternate assessments measure 
grade-level standards for the 
grade in which students are 
enrolled. 


STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 


5.3 Note: This answer addresses both students with disabilities and LEP. 

All students with disabilities in Montana public schools as defined under section 602(3) of the 
Individuals with Disabilities Education Act (IDEA) amendments of 1997 and limited English 
proficiency (LEP) participate in the Montana Comprehensive Assessment Program. The 
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Individualized Education Program (IEP) Team determines how students with disabilities will 
participate in the statewide assessment program. A team of educators familiar with the language 
abilities of LEP students determines how LEP students will participate. These requirements are 
contained in ARM 10.56.101, the assessment rules of the Board of Public Education found in the 
Administrative Rules of Montana (ARM). Most students with disabilities and all LEP students 
are included in the regular assessment with the use of allowable accommodations. Only for 
those students with significant cognitive disabilities not able to participate in the regular 
assessment, the Montana CRT Alternate Assessment Program ( based on alternate achievement 
standards ) yields reading and mathematics assessment results for inclusion in AYP 
determinations, utilizing Montana’s approved exception to the 1% rule. 

The achievement standards for Montana’s Alternate Assessment Program for students with 
disabilities and LEP are the same achievement standards applied to all students. Unlike many 
states, Montana chose not to establish an alternate assessment that measured performance of 
students with disabilities or LEP against a separate set of standards. Through a process of 
teacher rating scales, students are assessed on their proficiency by measuring their competence 
utilizing the performance goals and indicators as established in the state standards. 

As a result of establishing a system that relies on the standards based performance descriptors 
for the alternate assessment, there is a direct one - to - one relationship between the state 
standards and the alternate assessment. This methodology not only allows the direct 
correspondence between the student’s measures of proficiency toward the state standards, but 
also permits direct measures of the student’s performance at all levels of proficiency, novice 
through advanced. 

All of the required subgroups, including students with disabilities and LEP, who are enrolled in a 
school for a full academic year will be included in the performance measures that determine 
AYP for that school. All of the required subgroups, including students with disabilities and LEP, 
who are enrolled in a district for a full academic year will be included in the performance 
measures that determine AYP for that district. (Note: except for LEP students who are in the 
first year in a United States school as noted in Section 5.4.) 

Also, in making AYP determinations for schools and districts that otherwise would not make 
AYP solely on the basis of the students with disabilities subgroup for assessments conducted 
during the 2005-2006 school year, Montana calculates the percent proficient for each such 
school’s or district’s students with disabilities subgroup as follows: We divide 2.0% by the 
percentage of students in the State of Montana who have disabilities and round it to the 
nearest whole number. This number equals 17%. We then calculate for each covered school 
or district the sum of the actual percent of proficient scores for the students with disabilities 
subgroup and the proxy percent of 17%. This approach is consistent with transition option 1, 
as transmitted to the States by the U.S. Department of Education on May 10, 2005. 
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CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


5.4 How are students with limited 
English proficiency included in 
the State’s definition of 
adequate yearly progress? 


All LEP student participate in 
statewide assessments: 
general assessments with or 
without accommodations or a 
native language version of the 
general assessment based on 
grade level standards. 


LEP students are not fully 
included in the State 
Accountability System. 




State demonstrates that LEP 
students are fully included in 
the State Accountability 
System. 





STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



5.4 See answer 5.3 above. In the development of Montana’s accountability system LEP students 
will be included in the determination of adequate yearly progress in the same manner as all 
other students. 

Montana requires students who have been identified as limited English proficient (LEP) to 
participate in the statewide assessment program. (Administrative Rules of Montana ARM 
10.56.101) Coding on the statewide assessment provides for the identification of LEP students. 
Scores for LEP students are reported and disaggregated at the school, district, and state level. 
Accommodations or the Montana Alternative Assessment Program are provided for students as 
needed, who have received fewer than three years of instruction in English. LEP students who 
are enrolled in a United States school for the first school year will be tested and included in 
participation rates but the scores will not be used in proficiency for AYP determinations. 

All of the required subgroups, including students with LEP, who are enrolled in a school for a 
full academic year will be included in the performance measures that determine AYP for that 
school. All of the required subgroups, including students with LEP, who are enrolled in a 
district for a full academic year will be included in the performance measures that determine 
AYP for that district. 

In addition, Montana will permit the inclusion of test scores of LEP students in the LEP 
subgroup for AYP for up to two years beyond their transition from LEP status. 
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CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 

NOT MEETING REQUIREMENTS 


5.5 What is the State's 

definition of the minimum 
number of students in a 
subgroup required for 
reporting purposes? For 
accountability purposes? 


State defines the number of 
students required in a subgroup 
for reporting and accountability 
purposes, and applies this 
definition consistently across the 
State. 4 

Definition of subgroup will result in 
data that are statistically reliable. 


State does not define the required 
number of students in a subgroup 
for reporting and accountability 
purposes. 

Definition is not applied 
consistently across the State. 

Definition does not result in data 
that are statistically reliable. 



STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



5.5 Reporting: For reporting purposes there must be at least 10 students in any student 
subgroup. 

Accountability: The State faces unique problems in the design and implementation of an 
accountability system that will meet the requirements of AYP, particularly with respect to the 
statistical significance of sample sizes. Montana is extremely rural and covers a very large 
geographical area. This results in the state having a large number of very small schools and 
districts. 

The following table illustrates this characteristic clearly: 

Enrolled School District 

2003 Count Count 

<=400 402 24-8 

-^=00 322 443 

-^=23 1-84 93 

-^=40 72 5# 





-1 





-tt 

1 


— 1 
^--500 


J 

46 


1 

63 


>=1000 


1 U 

44 


U J 

34 



Well over half of Montana’s 864 schools have enrollments under 100, and the same pattern 
applies to districts. 



4 The minimum number is not required to be the same for reporting and accountability. 
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The impact of this on our AYP determination is that any arbitrary number the state might set as a 
lower threshold of statistical significance results in a very large number of schools falling below 
the limit for analysis. For example, if we were to choose 25 as a minimum sample size, only a 
fraction of one percent of our schools would qualify for analysis of the students with disabilities 
subgroup for student achievement. 

In view of the large proportion of schools that a) fall below even an admittedly low limit for 
purposes of statistical significance, and b) are often those most in need of assistance for a variety 
of other reasons, the state system for measuring AYP must be designed with explicit attention to 
this group. For these reasons, Montana chooses not to select a single arbitrary number 
representing statistical significance, but will employ an AYP determination based on confidence 
intervals at the .95 level of confidence. A one tailed method will be utilized. 

Montana will use a minimum of 30 for the “ All Students” group and a minimum of 40 for 
subgroups in making AYP determinations - including with regard to both participation rate 
and percent proficient in reading and math. (Participation rate in the “ All students ” group 
under 40 or subgroups under 40 will be passing with an actual enrollment minus no more 
than two not tested .) 

Montana will use a 95 percent confidence interval in determining AYP, only with regard to the 
determination of percent proficient in reading and math (and not with regard to either 
participation rate or the other academic indicators ). Moreover, beginning with testing data 
from spring 2005, Montana will use a limited, smaller, 75 percent confidence interval for 
“ safe harbor ” determinations, only where the given subgroup, school, or district has shown 
positive progress in reducing from the prior year the percentage of students scoring below 
proficiency (and the only question is whether that progress constitutes a 10 percent reduction). 
In addition, Montana will use a 99 percent confidence interval only as a filter for determining 
general inclusion of schools and districts in Montana’s broader state accountability process. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 

NOT MEETING REQUIREMENTS 


5.6 How does the State 
Accountability System 
protect the privacy of 
students when reporting 
results and when 
determining AYP? 


Definition does not reveal 
personally identifiable 
information. 5 


Definition reveals personally 
identifiable information. 



STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



5 The Family Education Rights and Privacy Act (FERPA) prohibits an LEA that receives Federal funds 
from releasing, without the prior written consent of a student’s parents, any personally identifiable 
information contained in a student’s education record. 
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5.6 The State neither publishes nor releases any data pertaining to school performance or other 
matters for any group or subgroup with fewer than 10 members. The SEA’s Web site allows any 
interested person to look up assessment and other data by school and by subgroup, but masks any 
results with counts below this limit. 

Individual student results are never reported to the public. In order to assure that individual 
students cannot be identified, school results are not publicly reported or displayed when the 
number of students in a subgroup is less than 10. Asterisks will be used on the Montana Report 
Card when data has been suppressed. 

Results greater than 95% will be reported as '>95%'’ and results less than 5 % will be reported as 
“<5%” in order to prevent the reporting of information that would violate the privacy of 
individual students. 

The State does maintain a password-protected version of the query utility for use by persons in 
the schools with a legitimate educational interest in accessing their own data for smaller groups. 

A given school or district, however, cannot access any data through this utility other than their 
own. 



PRINCIPLE 6. State definition of AYP is based primarily on the State’s academic 
assessments. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


6.1 How is the State’s 
definition of adequate 
yearly progress based 
primarily on academic 
assessments? 


Formula for AYP shows that 
decisions are based primarily on 
assessments. 6 

Plan clearly identifies which 
assessments are included in 
accountability. 


Formula for AYP shows that 
decisions are based primarily on 
non-academic indicators or 
indicators other than the State 
assessments. 


STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



6 State Assessment System will be reviewed by the Standards and Assessments Peer Review Team. 
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6.1 Reading and mathematics assessment scores are the predominant determinant of AYP. 

While the required additional academic indicators (NCLB Section 1111 (b)(2)(C)(vi)) are 
part of the AYP determination, in determining whether each subgroup, school, and district, 
as well as the state overall meets the annual measurable objectives, Montana will calculate 
the percent of the tested students who achieve the proficient level or higher, examine 
participation rates, examine the other indicators, and employ the safe harbor provision. 

To meet or exceed AYP, all student subgroups are required to meet the state’s definition of 
proficient for reading and mathematics. 

The assessments that will be used to determine AYP calculations for schools and LEAs in Montana 
are designated by “X” and on the following chart once all assessments are in place for 2005 - 2006 
school year: 



Chart 7. Montana’s Accountability Assessments 





MontCAS, Phase 2 


MT Alternate Assessment Program 


Grade 


R 


M 


R 


M 


K 










1 










2 










3 


X 


X 


X 


X 


4 


X 


X 


X 


X 


5 


X 


X 


X 


X 


6 


X 


X 


X 


X 


7 
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X 


X 


X 


X 


9 










10 


X 


X 


X 


X 


11 










12 











The same performance level standards will be applied to public schools and LEAs, 
disaggregating the data into the federally-defined subgroups to determine the minimum percent 
of students at or above the state performance level of proficient. 
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PRINCIPLE 7. State definition of AYP includes graduation rates for public High 
schools and an additional indicator selected by the State for public Middle and 
public Elementary schools (such as attendance rates). 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 

NOT MEETING REQUIREMENTS 


7.1 What is the State definition 
for the public high school 
graduation rate? 


State definition of graduation rate: 

• Calculates the percentage 
of students, measured 
from the beginning of the 
school year, who graduate 
from public high school 
with a regular diploma (not 
including a GED or any 
other diploma not fully 
aligned with the state’s 
academic standards) in 
the standard number of 
years; or, 

• Uses another more 
accurate definition that 
has been approved by the 
Secretary; and 

• Must avoid counting a 
dropout as a transfer. 

Graduation rate is included (in the 
aggregate) for AYP, and 
disaggregated (as necessary) for 
use when applying the exception 
clause 7 to make AYP. 


State definition of public high 
school graduation rate does not 
meet these criteria. 



STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



7 See USC 631 l(b)(2)(I)(i), and 34 C.F.R. 200.20(b) 
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7.1 Montana currently only gathers dropout data based upon an event rate adapted from the 
National Center for Education Statistics (NCES) at the U.S. Department of Education. The State 
will be developing a new reporting system for collecting, reporting and monitoring graduation 
rate based upon the following criteria: 

Montana’s initial threshold for graduation rate will be calculated in June 2003 once the data and 
method have been validated. 

Montana’s graduation rate is an estimated cohort group rate that is calculated by the method 
recommended by the NCES: 



Where: 



St /(c + g t + d 1 t + d 1 (t _ ]} + d (u2) + (f (t . 3) ) 

g = # of graduates receiving a standard high school diploma in standard # of years 
c = completers of high school by other means 
t = year of graduation 
d = dropouts 

12, 11, 10, 9 = class level 



All students graduating in Montana receive a standard high school diploma and will be counted 
as a graduate. Students receiving a GED are not included as graduates when calculating 
graduation rates. Montana’s definition of a dropout is consistent with the requirements of the 
NCES Common Core of Data (CCD) reporting. According to Montana’s definition, a dropout is 
an individual who: 

• Was enrolled in school on the date of the previous year October 
enrollment count or at sometime during the previous school year and was 
not enrolled on the date of the current school year October count; or 

• Was not enrolled at the beginning of the previous school year but was 
expected to enroll and did not reenroll during the year (“no show”) and 
was not enrolled on the date of the current school year October count; and 

• Has not graduated from high school or completed a state- or district- 
approved high school educational program; and 

• Has not transferred to another school, been temporarily absent due to a 
school-recognized illness or suspension, or died. (Montana Dropout 
Statistics Collector’s Handbook, 
http://www.opi.state.mt.us/pdf/acd/dohandbook.pdf) . 

Since graduate and dropout data are not reported until after the beginning of the school year, the 
graduation rate from the previous year will be used (e.g., 2002 graduation rate in 2003) for all 
AYP determinations. For purposes of AYP (other than “safe harbor”) the calculation of the 
graduation rate will apply to the school and district level, but not to the subgroup level. Schools 
and districts that achieve or exceed the threshold for the graduation rate, as well as those that are 
below the threshold but improve their graduation rate when compared to the previous year, will 
have met the other academic indicator for purposes of calculating AYP. 
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In addition to being part of the definition of AYP, schools/districts will be required to meet the 
graduation rate threshold or improve their graduation rate as a requirement for the “safe harbor” 
provision (subgroups that fail to meet AYP standards but succeed in reducing the proportion of 
students who are not proficient by at least ten percent). Currently, the State collects graduation 
and dropout data by the required disaggregations for the NCES Common Core of Data (CCD) 
reporting (racial/ethnic origin and gender). The State will begin collecting graduate/dropout data 
for the subgroups currently not collected (special education, limited English proficient, poverty) 
during the October 2003 2004 Annual Data Collection. However, as can be seen from the 
graduation rate formula above, it will take four years for Montana to be able to calculate these 
disaggregated graduation rates. Until that time, schools/districts will be required to provide to 
the State graduate/dropout data for subgroups not collected by the State to enact “safe harbor” 
provision. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 

NOT MEETING REQUIREMENTS 


7.2 What is the State’s 
additional academic 
indicator for public 
elementary schools for the 
definition of AYP? For 
public middle schools for 
the definition of AYP? 


State defines the additional 
academic indicators, e.g., 
additional State or locally 
administered assessments not 
included in the State assessment 
system, grade-to-grade retention 
rates or attendance rates. 8 

An additional academic indicator 
is included (in the aggregate) for 
AYP, and disaggregated (as 
necessary) for use when applying 
the exception clause to make 
AYP. 


State has not defined an 
additional academic indicator for 
elementary and middle schools. 



8 NCLB only lists these indicators as examples. 
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STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



7.2. Montana schools have reported attendance rates annually for the last nine years. The 
Montana Office of Public Instruction has selected attendance rate as the additional academic 
indicator for calculating AYP for elementary and middle schools. The calculation for the student 
attendance rate is based on the following: 

The Montana Office of Public Instruction collects “present” and “absent” information at two 
points in the school year: October and February. This process establishes the “average daily 
attendance” or ADA as required by Administrative Rules of Montana - 10.15.103(3) and 
10.20.102(6). Currently the data are not disaggregated at the state level. By October 2004, 
Montana will collect and analyze required categories for disaggregation for use in Safe Harbor 
decisions and for display on school, district, and state Report Cards. Until that time, 
schools/districts will be required to provide to the State attendance data for subgroups not 
collected by the State to enact “safe harbor” provision. 

The Montana State Superintendent will establish the attendance rate standard. Schools will be 
considered as having met AYP if they meet or exceed the standard or if they have made 
improvement toward the standard. Note: This standard has now been set at 80 percent. 

For the AYP determination, the attendance rate calculation will be used for accountability at the 
school/LEA levels, but will not be used for each subgroup. However, for schools/LEAs that 
must use the “safe harbor” provision to meet AYP for the achievement indicator, the attendance 
rate standard must then be met by the subgroup(s) that failed to meet AYP on the assessment 
standards. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 

NOT MEETING REQUIREMENTS 


7.3 Are the State’s academic 


State has defined academic 


State has an academic indicator 


indicators valid and 


indicators that are valid and 


that is not valid and reliable. 


reliable? 


reliable. 


State has an academic indicator 




State has defined academic 


that is not consistent with 




indicators that are consistent with 
nationally recognized standards, if 


nationally recognized standards. 




any. 


State has an academic indicator 
that is not consistent within grade 
levels. 
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STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



7.3 These defined academic indicators will be valid and reliable as demonstrated by the use 
of clear definitions for data elements and the use of statewide systems for data collection 
(ADC and MAEFAIRS). The Montana Office of Public Instruction reviews the data and 
monitors the databases to verify accuracy of the data. The graduation rate calculation is 
consistent with the NCES completer formula and the attendance rate calculation is the same 
as that used for Impact Aid purposes. 



50 





CONSOLIDATED STATE APPLICATION ACCOUNTABILITY WORKBOOK 



PRINCIPLE 8. AYP is based on reading/language arts and mathematics 
achievement objectives. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


8.1 Does the state measure 
achievement in 
reading/language arts and 
mathematics separately for 
determining AYP? 


State AYP determination for 
student subgroups, public 
schools and LEAs separately 
measures reading/language arts 
and mathematics. 9 

AYP is a separate calculation for 
reading/language arts and 
mathematics for each group, 
public school, and LEA. 


State AYP determination for 
student subgroups, public 
schools and LEAs averages or 
combines achievement across 
reading/language arts and 
mathematics. 


STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 


8.1 Yes. Under the current system the state measures performance levels of reading and 
mathematics separately at the elementary, middle, and high school level. These are the data that 
will be used to identify schools and districts not making AYP using the March 2003 results. 

For the new assessments, AYP determinations will continue at the school, district, and state 
levels and for subgroups therein. The calculation will examine separately the proportion of 
student proficiency and participation in reading and mathematics. The State will calculate 
separately for reading and math the percent of tested students who achieve the proficient level or 
above, as well as below, for the “Safe Harbor” provision. For tests under development the 
performance levels and participation rate also will be measured separately for reading and 
mathematics at the different levels and include the subgroups. 



9 If the state has more than one assessment to cover its language arts standards, the State must create a 
method for including scores from all the relevant assessments. 
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PRINCIPLE 9. State Accountability System is statistically valid and reliable. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


9.1 HowdoAYP 

determinations meet the 
State’s standard for 
acceptable reliability? 


State has defined a method for 
determining an acceptable level of 
reliability (decision consistency) 
for AYP decisions. 

State provides evidence that 
decision consistency is (1) within 
the range deemed acceptable to 
the State, and (2) meets 
professional standards and 
practice. 

State publicly reports the estimate 
of decision consistency, and 
incorporates it appropriately into 
accountability decisions. 

State updates analysis and 
reporting of decision consistency 
at appropriate intervals. 


State does not have an 
acceptable method for 
determining reliability (decision 
consistency) of accountability 
decisions, e.g., it reports only 
reliability coefficients for its 
assessments. 

State has parameters for 
acceptable reliability; however, 
the actual reliability (decision 
consistency) falls outside those 
parameters. 

State’s evidence regarding 
accountability reliability (decision 
consistency) is not updated. 


STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 


9.1 In accordance with the terms of the Compliance Agreement, Montana will provide the 
process that creates evidence that the State Accountability System is reliable no later than 
November 30, 2004. The reliability of accountability system determinations will be assured 
through: 

• uniform averaging of scale scores across grade levels within the school (where 
possible) and LEA to produce a single school or LEA score; 

• multiple year averages using no more than 3 years of data to determine reading and 
mathematics proficient levels of performance for rating public schools. 
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• statistical tests to support the use of confidence intervals; 

• methods for determining an acceptable level of reliability for consistent decisions for 
two years for purposes of research and investigation; and 

• Safe Harbor provision and evidence that this rule increases reliability of decisions 
about schools. 

Montana will make AYP determinations for all public schools and districts based 
primarily on state assessment data. Montana proposes to make AYP determinations 
for small schools where the total number of test scores is below the minimum number 
of 30 in the all students group based on a tiered process, using the state 
assessment/AYP data for each school and district along with a broader, qualitative 
review of school and subgroup performance data and other information related to 
student achievement where necessary to ensure the most valid and reliable AYP 
determinations ( Small Schools Process ). 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


9.2 What is the State's process 
for making valid AYP 
determinations? 


State has established a process 
for public schools and LEAs to 
appeal an accountability decision. 


State does not have a system for 
handling appeals of accountability 
decisions. 



STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 
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9.2 Montana’s accountability process is designed for construct validity and consequential 
validity. 

Construct validity: 

Components of the accountability system include content standards, assessment, data collection 
and reporting, the identification of schools for improvement, providing rewards and sanctions, 
and technical assistance to the purpose of improving Montana schools. The following 
components work in harmony to accomplish school improvement: 

Assessment: 

The MontCAS, Phase 2 and Montana Alternate Assessment results are the primary 
indicators on which AYP determinations are made for public schools and LEAs. 

Students with disabilities and LEP students may receive accommodations and 
modifications on the MontCAS, Phase 2 or be assessed through Montana Alternate 
Assessment, if they meet the criteria as outlined in guidance from the state and 
determined by the IEP or LEP Team. Our assessment system will provide technical data 
to include: 
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-Evidence of reliability and validity 
-Internal and external alignment studies 
-Internal and external bias studies 
-Systemic procedures for quality checks 

Data Collection: 

Enrollment information about students and their membership in specific subgroups is 
determined at the school level through programs provided by OPI. Each school and LEA 
must verify the accuracy of the information contained in the files submitted to OPI for 
enrollment and group identification purposes. 

Policies that will be incorporated into the OPI State and Federal Grants Handbook require 
that all students enrolled for a full academic year be included in the accountability 
formula. This policy will also outline the procedures for instituting and maintaining a 
valid system. 

Consequential validity: 

Reliable assessments aligned with content standards will result in accurate identification of 
schools and LEAs in need of improvement. Accurate data collection and reporting will support 
the inferences drawn from the accountability system. Schools and LEAs will have access to an 
appeals procedure following preliminary identification. 

In order to increase the validity of accountability decisions, the OPI State and Federal Grants 
Handbook will include the following Appeals Process: 

1) The OPI determines preliminary identification of all schools and LEAs that have not met 
AYP according to the state criteria and sends notifications. 

2) Within 30 days of preliminary identification, the agency (LEA/school) reviews its data 
and may challenge its identification. The agency (LEA/school) not meeting AYP may 
appeal its status and provide evidence to support the challenge to the OPI. 

3) No later than 30 days after preliminary identification, the OPI reviews the appeal and 
makes a final determination of identification for school improvement. {Section 1116 (b) 
(2) (A) (B) (C) } 
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A valid and reliable accountability system has been designed for the 2003-04 new assessment 
program that includes the requirements of NCLB. The new accountability system will be 
designed to create the most advantageous balance of 1) reliable results, 2) public confidence in 
the results, 3) including all public schools in the accountability formula, and 4) capacity building 
and development of resources to serve Montana students and schools. 

As the new Montana Accountability System is implemented, Montana will regularly examine the 
validity and reliability of the data related to the determination of AYP and decision consistency 
for holding public schools and LEAs accountable within this system. Updated analysis and 
reporting of decision consistency will be shared with the public at appropriate intervals. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


9.3 How has the State planned 
for incorporating into its 
definition of AYP 
anticipated changes in 
assessments? 


State has a plan to maintain 
continuity in AYP decisions 
necessary for validity through 
planned assessment changes, 
and other changes necessary to 
comply fully with NCLB. 10 

State has a plan for including new 
public schools in the State 
Accountability System. 

State has a plan for periodically 
reviewing its State Accountability 
System, so that unforeseen 
changes can be quickly 
addressed. 


State’s transition plan interrupts 
annual determination of AYP. 

State does not have a plan for 
handling changes: e.g., to its 
assessment system, or the 
addition of new public schools. 



10 Several events may occur which necessitate such a plan. For example, (1) the State may need to 
include additional assessments in grades 3-8 by 2005-2006; (2) the State may revise content and/or 
academic achievement standards; (3) the State may need to recalculate the starting point with the 
addition of new assessments; or (4) the State may need to incorporate the graduation rate or other 
indicators into its State Accountability System. These events may require new calculations of validity and 
reliability. 
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STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



9.3 As a Title I compliance agreement state, Montana will use its current assessment and 
accountability system as the basis for adequate yearly progress decisions during the transition 
period of 2002-03. Beginning in 2003-04, Montana will administer the new assessments to 
determine AYP for Montana schools, districts, and the state as described previously in this 
document. 

The identification of schools and districts (by all subgroups including all students) not achieving 
an average score of 45 NCE on the 2002-03 assessment and the percent of students who meet or 
exceed the proficient level definition on the 2003-04 assessment, as well as participation rates 
and the other indicators, will be used to continuously identify schools and districts in need of 
improvement during the transition from the old assessment program to the new assessment 
program. 

The State has a plan for including new public schools in the State Accountability System. New 
public schools will be included in each assessment and held to the same AYP standards as all 
other public schools. 

The State periodically will review its accountability system. 
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PRINCIPLE 10. In order for a public school or LEA to make AYP, the State 
ensures that it assessed at least 95% of the students enrolled in each subgroup. 



CRITICAL ELEMENT 



1 0.1 What is the State's method for 
calculating participation rates 
in the State assessments for 
use in AYP determinations? 



EXAMPLES FOR 
MEETING 
REQUIREMENTS 



State has a procedure to 
determine the number of absent 
or untested students (by 
subgroup and aggregate). 

State has a procedure to 
determine the denominator 
(total enrollment) for the 95% 
calculation (by subgroup and 
aggregate). 

Public schools and LEAs are 
held accountable for reaching 
the 95% assessed goal. 



EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 



The state does not have a 
procedure for determining the 
rate of students participating in 
statewide assessments. 

Public schools and LEAs are 
not held accountable for testing 
at least 95% of their students. 



STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 



10.1 For use with the current system in March 2003 and the new system beginning April 2004, 
Montana has developed a verification form (see Appendix C) that each school completes during 
the testing cycle. By school and grade level tested, the verification form validates the test day 
enrollment, changes in that enrollment from the official spring enrollment on the first Monday in 
February, and the number of students tested. Any changes in numbers that affect participation 
rate are explained. 

For 2003 AYP determinations, using the spring enrollment, the verification forms, and answer 
documents coded for demographics of students not tested, the state can determine the 
denominator (total enrollment) for the 95% calculation for all subgroups. 

The students identified as enrolled in the tested grades on the date of the verification form are 
those students that are expected to have taken the statewide assessment or Montana Alternate 
Assessment. 
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Once the tests are scanned and scored, the file of students tested is matched against the verified 
enrollment file to determine who did and who did not take the test for each academic subject. 
The number of tested students divided by the number of enrolled students is the percent tested. 
The number enrolled but not tested divided by the number enrollment is the percent not tested. 

The calculation for participation rate is: 

(students not tested/students enrolled) x 100 = % not tested 

(students tested/students enrolled) x 100 = % tested 

Participation rate will be determined for each subject and for each subgroup. Students with 
invalid assessment scores are counted as non-participants for the purpose of calculating 
participation rate. 

Montana will allow 95% participation rate to be calculated based on averaged data and will 
permit exceptions regarding student participation in limited cases of medical emergency, 
which may be raised by schools or districts on appeal of AYP determinations. 



CRITICAL ELEMENT 


EXAMPLES FOR 
MEETING REQUIREMENTS 


EXAMPLES OF 
NOT MEETING 
REQUIREMENTS 


10.2 What is the State's policy 
for determining when the 
95% assessed 
requirement should be 
applied? 


State has a policy that 
implements the regulation 
regarding the use of 95% 
allowance when the group is 
statistically significant according 
to State rules. 


State does not have a procedure 
for making this determination. 


STATE RESPONSE AND STATE ACTIVITIES FOR MEETING REQUIREMENTS 


10.2 The participation rate is determined as having been achieved by either verifying that the 
number of students tested is equal to or greater than 95% of the actual enrollment or equal to or 
greater than the actual enrollment minus two for schools/districts with less than 40 students. 
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Appendix A 

Required Data Elements for State Report Card 



1111(h)(1)(C) 

1 . Information, in the aggregate, on student achievement at each proficiency level on the State academic 
assessments (disaggregated by race, ethnicity, gender, disability status, migrant status, English 
proficiency, and status as economically disadvantaged, except that such disaggregation shall not be 
required in a case in which the number of students in a category is insufficient to yield statistically reliable 
information or the results would reveal personally identifiable information about an individual student. 

2. Information that provides a comparison between the actual achievement levels of each student 
subgroup and the State’s annual measurable objectives for each such group of students on each of the 
academic assessments. 

3. The percentage of students not tested (disaggregated by the student subgroups), except that such 
disaggregation shall not be required in a case in which the number of students in a category is insufficient 
to yield statistically reliable information or the results would reveal personally identifiable information 
about an individual student. 

4. The most recent 2-year trend in student achievement in each subject area, and for each grade level, 
for the required assessments. 

5. Aggregate information on any other indicators used by the State to determine the adequate yearly 
progress of students in achieving State academic achievement standards disaggregated by student 
subgroups. 

6. Graduation rates for secondary school students disaggregated by student subgroups. 

7. Information on the performance of local educational agencies in the State regarding making adequate 
yearly progress, including the number and names of each school identified for school improvement under 
section 1116. 

8. The professional qualifications of teachers in the State, the percentage of such teachers teaching with 
emergency or provisional credentials, and the percentage of classes in the State not taught by highly 
qualified teachers, in the aggregate and disaggregated by high-poverty compared to low-poverty schools 
which (for this purpose) means schools in the top quartile of poverty and the bottom quartile of poverty in 
the State. 



60 




CONSOLIDATED STATE APPLICATION ACCOUNTABILITY WORKBOOK 



Appendix B 




STATE OF MONTANA 
Office of Public Instruction 
INVITATION FOR BID 

(THIS IS NOT AN ORDER) 



IFB Number: 


IFB Title: 


35060407W 


Web Report Cards 


IFB Due Date and Time: 






April 18, 2003 


Number of Pages: Eight 


2:00 p.m. local time 





ISSUING AGENCY INFORMATION 






Procurement Officer: 




Issue Date: 


Sioux Roth, Purchasing 




April 7, 2003 


OFFICE OF PUBLIC INSTRUCTION 
P O Box 202501 
1227 11- Avenue 


Phone: (406) 444-4404 


Helena, MT 59620-2501 


Fax: (406) 444-1369 




Website: www.opi.state.mt.us 



INSTRUCTIONS TO BIDDERS 


COMPLETE THE INFORMATION BELOW 
AND RETURN THIS PAGE WITH YOUR BID 
AND ANY REQUIRED DOCUMENTS TO THE 
ADDRESS LISTED ABOVE UNDER "ISSUING 
AGENCY INFORMATION." 


Mark Face of Envelope/Package: 

IFB Number: WEB REPORT CARDS, P O 
#35060407W 

IFB Due Date: 04/18/03, 2:00 P.M. LOCAL TIME 


Special Instructions: 


Alternate Bids: 

Vendors may submit alternate bids (a bid on supplies other than specified). Alternate bids are considered only if the vendor is the 
lowest responsible vendor on their primary bid. Bids must be clearly identified as "Primary" and "Alternate." 
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BIDDERS MUST COMPLETE THE FOLLOWING 




Payment Terms: Payment upon 
completion 


Delivery Date: 


Bidder Name/Address: 


Authorized Bidder Signatory: 

(Please print name and sign in ink) 


Bidder Phone Number: 


Bidder FAX Number: 


Bidder Federal I.D./Social Security 
Number: 


Bidder E-mail Address: 


IMPORTANT: SEE STANDARD TERMS AND CONDITIONS 



Agency: Montana Office of Public Instruction 
Project: Web Report Cards 



Successful bidder will 

• Successfully complete project for less than $25,000. 

• Have a proven, documented Project Management model. 

• Have a proven, documented Software Development model. 

• Have successful experience working with state or national level education agency. 

• Demonstrate product using Scalar Vector Graphics (SVG) technology that is similar to 
Wyoming’s School Report Cards (http://www.kl2.wv.us/pls/stats/esc.show menu) 

• Demonstrate ability to create SVG graphics from XML based queries 
(http : //www . w3 . org/Graphic s/S V G/O vervie w .htm8 ) . 
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Project Description 

The contractor will develop web-based School Report cards for Montana schools, districts, and 
state. The contractor will train and provide initial support to OPI staff in the creation and 
maintenance of report cards. The report cards summarize assessment, demographic, and other 
indicators in accordance with the requirements of the No Child Left Behind Act (NCLB). 
Montana contains 988 schools and 465 districts. 

The web-based report cars are a preferred solution because they simply and elegantly display 
school indicators on the web. The method of using XML and SVG provides a relatively 
straightforward, open source means to produce school report cards that can be used in reports by 
local districts as well as accessed by the general public. This solution works with OPI’s existing 
deployment strategy. Wyoming has done most of the work of determining the formatting of 
school indicators in a way that is easily understandable. OPI is attempting to leverage 1.5 years 
of Wyoming effort. OPI wants a contractor familiar with education business context, relational 
databases, XML, SVG, and efficient software development process to expedite the report card 
development. 



Montana Information Technology Plan 

The project satisfies these goals of the Montana Information Technology Plan: 

Objective #1 - Support schools so that all students can achieve high standards. 

1.1 Provide assistance to Montana educators in the efficient and effective use of resources. The 
Information Technology Division will continue to provide electronic information resources (Web 
Report Cards) to OPI and field staff. 

Objective #2 - Assess and communicate the quality and achievements of K-12 education 

2.1 Develop and disseminate a Montana Statewide Educational Profile that presents information 
about our K-12 public schools and students across a broad range of indicators of quality. IT staff 
will provide portions of this profile on line and in real time so that the public and the legislature 
know the status of schools. Measure: the time to complete and number of profile features 
available. 

Objective #3 - Provide access to and management of information and data related to K-12 school 

improvement 

3.1 Develop and disseminate the Montana Statewide Education Profile and the local 
district formats. The web staff will deliver via the Internet. Measure: number of 
downloads or accesses. 

3.2 Assist Montana educators in the development of local education profiles. The OPI 
professional staff and IT staff will work together to training staff, publish, and document 
local profiles. Measure: number of profiles available via OPI’s web site; field feedback of 
process; federal audit findings. 

3.5 Develop and enhance electronic systems for Montana educators to report and retrieve 
information and data related to K-12 education. IT staff will make summarized 
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information available on the web site for as many collections as possible. Measure: 
percent of collection reporting summarized data back to the OPI web site. 



Project/Task Objectives and Requirements 

In General, the contractor and client agree to: 

• Great customer service to OPI 

• Results in a product similar to Wyoming’ s Report card 

• Development process follows a successfully proven model 

• The OPI team will meet in advance of contractor’s initial visit to hammer out a common 
vision 

• The contractor will visit Helena for 2-4 days to lead us through our joint development 
process 

• OPI staff will generate the data for the charts and graphs in accordance to the agreed 
upon specifications 

Scope of Work, Deliverables and Acceptance Criteria 

Contractor shall provide Services and staff, and otherwise do all things necessary for or 
incidental to the performance of work, as set forth below: 

Phase 1: 

Initial trip to Helena and meeting [Week of April 28 th ] 

3-4 days on site to nail down analysis on what reports should look like. 

Assign project roles and explain communication structure. 

Set expectation that this runs currently to run through to completion. 

Draw out all reports. 

Get to work on detail design for each screen. 

Identify products to be delivered (4-5 main SVG templates) [Week of April 28 th ] 

Identify number of reports 

Identify specific information on reports 

Identify number of combinations (i.e. 988 schools x 5 school reports = 4,940) 
Identify timelines: 

2-3 month consecutive month commitment for team members [April 21 - July 14] 
Design web pages for Report Card site [May 1-15] 

Customizations to reports to distinguish from Wyoming [May 15-June 30] 
Mentoring OPI staff in methods and modifications [May 15-July 14] 

Implementation [June 1 - July 14] 

Technical documentation [On-going] 

Visually shows system 
Provides help in modifying system 

Phase 2 (if OPI wants to continue after Phase 1): 

Create means to generate PDF pages with annotations 
Option: Active reports ($3,000 -$4,000) 

Option: Java reports (mostly free) 
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Both Phase I and II will not exceed $25,000 totally 
Contractor shall produce: 

Results in a product similar to Wyoming’s Report card (see above web site) 

Charts and text that represents Montana’s school, district, and state status 
Web Report Card templates in SVG format as XLS style sheets 
Field level definitions needed by XLS style sheets. 

Consultation on efficient means to display web report cards 
Ability to cut and past SVG charts in page design programs used by newspapers 
Work with OPI staff to define specific report card content and presentation format. 
Training/mentoring for OPI’s programming and web staff in the means and methods of 
working with, designing, and creating SVG files. 

The following criteria will be used by the Agency to determine acceptance of the services and/or 
deliverables provided under this SOW. 

• All report templates work with OPI XML queries to generate web report cards 

o Reports show content agreed to by OPI and the contracting staff 
o Reports show content correctly (correct calculations) 
o Report generate quickly 

• Training is provided to OPI staff so OPI staff can understand, change, delete, and create 
their own SVG templates or modify existing templates. 

o OPI staff can demonstrate their knowledge by modifying templates 

Timeline and Period of Performance 

The period of performance for this project will start on April 28 and the work tasks are estimated 
to continue through July 14. The State has the right to extend or terminate this SOW at its sole 
discretion. 

Compensation and Payment 

Agency shall pay Contractor an amount not to exceed $25,000 dollars for the performance of all 
activities necessary for or incidental to the performance of work as set forth in this SOW. The 
contractor’s hourly compensation for services rendered shall be based on Contractor’s Prices as 
set forth in the Consulting and Services Agreement. 

Agency shall reimburse Contractor for travel and other expenses as identified in this SOW, or as 
authorized in writing, in advance by Agency. No payment of travel expenses will be made to 
Contractor for routine travel to and from Agency’s location. Contractor shall provide a detailed 
itemization of expenses as requested by Agency. The amount reimbursed to Contractor is 
included in calculating the “not to exceed” amount specified above. 



Contractor Staff, Roles and Responsibilities 



Role/Area 


Who 


Title 


E-mail 




Phone 


Project 

Manager 


To Be Named 


Senior Consulting 
Manager 






Programming 


To Be Named 


Programmer 
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Contact 










Technical 

Contact 


To Be Named 


Senior Technical 
Manager 







Additional Terms and Conditions Specific to this SOW 

All work and intermediate work products will be delivered to the Montana Office of Public Instruction. 

Team Guidelines 

One or two individuals focused on data side. 

People involved who understand the statistical reporting, so data is formatted correctly. 
Response needed by e-mail and/or phone within 24-hour from all team members. (Usually 
simple questions) 

Style and formatting leader for web report cards. 



Standard Terms and Conditions 



By submitting a response to this invitation for bid, request for proposal, limited 

solicitation, or acceptance of a contract, the vendor agrees to acceptance of the 
following Standard Terms and Conditions and any other provisions that are 
specific to this solicitation or contract. 



ACCEPTANCE/REJECTION OF BIDS, PROPOSALS, OR LIMITED SOLICITATION 
RESPONSES: The State reserves the right to accept or reject any or all bids, proposals, or 
limited solicitation responses, wholly or in part, and to make awards in any manner deemed in 
the best interest of the State. Bids, proposals, and limited solicitation responses will be firm for 
30 days, unless stated otherwise in the text of the invitation for bid, request for proposal, or 
limited solicitation. 

ACCESS AND RETENTION OF RECORDS: The contractor agrees to provide the department, 
Legislative Auditor, or their authorized agents, access to any records necessary to determine 
contract compliance (Mont. Code Ann. § 1 8-1 -118). The contractor agrees to create and retain 
records supporting the services rendered or supplies delivered for a period of three years after 
either the completion date of the contract or the conclusion of any claim, litigation, or exception 
relating to the contract taken by the State of Montana or third party. 

ALTERATION OF SOLICITATION DOCUMENT: In the event of inconsistencies or 
contradictions between language contained in the State's solicitation document and a vendor's 
response, the language contained in the State's original solicitation document will prevail. 
Intentional manipulation and/or alteration of solicitation document language will result in the 
vendor's disqualification and possible debarment. 
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ASSIGNMENT, TRANSFER AND SUBCONTRACTING: The contractor shall not assign, 
transfer or subcontract any portion of the contract without the express written consent of the 
department. (Mont. Code Ann. § 18-4-141.) 

AUTHORITY: The following bid, request for proposal, limited solicitation, or contract is issued 
under authority of Title 18, Montana Code Annotated, and the Administrative Rules of Montana, 
Title 2, chapter 5. 

COMPLIANCE WITH LAWS: The contractor must, in performance of work under the contract, 
fully comply with all applicable federal, state, or local laws, rules and regulations, including the 
Montana Human Rights Act, the Civil Rights Act of 1964, the Age Discrimination Act of 1975, 
the Americans with Disabilities Act of 1 990, and Section 504 of the Rehabilitation Act of 1 973. 
Any subletting or subcontracting by the contractor subjects subcontractors to the same 
provision. In accordance with section 49-3-207, MCA, the contractor agrees that the hiring of 
persons to perform the contract will be made on the basis of merit and qualifications and there 
will be no discrimination based upon race, color, religion, creed, political ideas, sex, age, marital 
status, physical or mental disability, or national origin by the persons performing the contract. 

CONFORMANCE WITH CONTRACT: No alteration of the terms, conditions, delivery, price, 
quality, quantities, or specifications of the contract shall be granted without prior written consent 
of the State Procurement Bureau. Supplies delivered which do not conform to the contract 
terms, conditions, and specifications may be rejected and returned at the contractor's expense. 

DEBARMENT: The contractor certifies that neither it nor its principals are presently debarred, 
suspended, proposed for debarment, declared ineligible, or voluntarily excluded from 
participation in this transaction (contract) by any governmental department or agency. If the 
contractor cannot certify this statement, attach a written explanation for review by the State. 

DISABILITY ACCOMMODATIONS: The State of Montana does not discriminate on the basis of 
disability in admission to, access to, or operations of its programs, services, or activities. 
Individuals, who need aids, alternative document formats, or services for effective 
communications or other disability-related accommodations in the programs and services 
offered, are invited to make their needs and preferences known to this office. Interested parties 
should provide as much advance notice as possible. 

FACSIMILE RESPONSES: Facsimile responses will be accepted for invitations for bids, small 
purchases or limited solicitations ONLY if they are completely received by the State 
Procurement Bureau prior to the time set for receipt. Bids, or portions thereof, received after the 
due time will not be considered. Facsimile responses to requests for proposals are ONLY 
accepted on an exception basis with prior approval of the procurement officer. 

FAILURE TO HONOR BID/PROPOSAL: If a bidder/offeror to whom a contract is awarded 
refuses to accept the award (PO/contract) or, fails to deliver in accordance with the contract 
terms and conditions, the department may, in its discretion, suspend the bidder/offeror for a 
period of time from entering into any contracts with the State of Montana. 

HOLD HARMLESS/INDEMNIFICATION: The contractor agrees to protect, defend, and save 
the State, its elected and appointed officials, agents, and employees, while acting within the 
scope of their duties as such, harmless from and against all claims, demands, causes of action 
of any kind or character, including the cost of defense thereof, arising in favor of the contractor's 
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employees or third parties on account of bodily or personal injuries, death, or damage to 
property arising out of services performed or omissions of services or in any way resulting from 
the acts or omissions of the contractor and/or its agents, employees, representatives, assigns, 
subcontractors, except the sole negligence of the State, under this agreement. 

LATE BIDS AND PROPOSALS: Regardless of cause, late bids and proposals will not be 
accepted and will automatically be disqualified from further consideration. It shall be solely the 
vendor's risk to assure delivery at the designated office by the designated time. Late bids and 
proposals will not be opened and may be returned to the vendor at the expense of the vendor or 
destroyed if requested. 

PAYMENT TERM: All payment terms will be computed from the date of delivery of supplies or 
services OR receipt of a properly executed invoice, whichever is later. Unless otherwise noted 
in the solicitation document, the State is allowed 30 days to pay such invoices. All contractors 
may be required to provide banking information at the time of contract execution in order to 
facilitate State electronic funds transfer payments. 

RECIPROCAL PREFERENCE: The State of Montana applies a reciprocal preference against a 
vendor submitting a bid from a state or country that grants a residency preference to its resident 
businesses. A reciprocal preference is only applied to an invitation for bid for supplies or an 
invitation for bid for nonconstruction services for public works as defined in section 18-2-401(9), 
MCA, and then only if federal funds are not involved. For a list of states that grant resident 
preference, see 

http://www.discoverinqmontana.com/doa/qsd/css/Resources/ReciprocalPreference.asp . 

REFERENCE TO CONTRACT: The contract or purchase order number MUST appear on all 
invoices, packing lists, packages and correspondence pertaining to the contract. 

REGISTRATION WITH THE SECRETARY OF STATE: Any business intending to transact 
business in Montana must register with the Secretary of State. Businesses that are incorporated 
in another state or country, but which are conducting activity in Montana, must determine 
whether they are transacting business in Montana in accordance with sections 35-1-1026 and 
35-8-1001, MCA. Such businesses may want to obtain the guidance of their attorney or 
accountant to determine whether their activity is considered transacting business. 

If businesses determine that they are transacting business in Montana, they must register with 
the Secretary of State and obtain a certificate of authority to demonstrate that they are in good 
standing in Montana. To obtain registration materials, call the Office of the Secretary of State at 
(406) 444-3665, or visit their website at http://www.sos.state.mt.us . 

SEPARABILITY CLAUSE: A declaration by any court, or any other binding legal source, that 
any provision of the contract is illegal and void shall not affect the legality and enforceability of 
any other provision of the contract, unless the provisions are mutually dependent. 

SHIPPING: Supplies shall be shipped prepaid, F.O.B. Destination, unless the contract specifies 
otherwise. 

SOLICITATION DOCUMENT EXAMINATION: Vendors shall promptly notify the State of any 
ambiguity, inconsistency, or error, which they may discover upon examination of a solicitation 
document. 
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TAX EXEMPTION: The State of Montana is exempt from Federal Excise Taxes (#81-0302402). 

TECHNOLOGY ACCESS FOR BLIND OR VISUALLY IMPAIRED: Contractor acknowledges 
that no state funds may be expended for the purchase of information technology equipment and 
software for use by employees, program participants, or members of the public unless it 
provides blind or visually impaired individuals with access, including interactive use of the 
equipment and services, that is equivalent to that provided to individuals who are not blind or 
visually impaired. (Mont. Code Ann. § 18-5-603.) Contact the State Procurement Bureau at 
(406) 444-2575 for more information concerning nonvisual access standards. 

TERMINATION OF CONTRACT: Unless otherwise stated, the State may, by written notice to 
the contractor, terminate the contract in whole or in part at any time the contractor fails to 
perform the contract. 

UNAVAILABILITY OF FUNDING: The contracting agency, at its sole discretion, may terminate 
or reduce the scope of the contract if available funding is reduced for any reason. (Mont. Code 
Ann. § 18-4-313 (3).) 

U.S. FUNDS: All prices and payments must be in U.S. dollars. 

VENUE: This solicitation is governed by the laws of Montana. The parties agree that any 
litigation concerning this bid, request for proposal, limited solicitation, or subsequent contract, 
must be brought in the First Judicial District in and for the County of Lewis and Clark, State of 
Montana, and each party shall pay its own costs and attorney fees. (Mont. Code Ann. § 18-1- 
401.) 

WARRANTIES: The contractor warrants that items offered will conform to the specifications 
requested, to be fit and sufficient for the purpose manufactured, of good material and 
workmanship and free from defect. Items offered must be new and unused and of the latest 
model or manufacture, unless otherwise specified by the State. They shall be equal in quality 
and performance to those indicated herein. Descriptions used herein are specified solely for the 
purpose of indicating standards of quality, performance and/or use desired. Exceptions will be 
rejected. 



Revised 3/03 
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Appendix C 

Student Participation Validation Form 

«SC» «School_Name» 
Grade: «Grade» 

Montana Comprehensive Assessment System, Phase 1 
Spring 2003 

Consistent with accountability requirements under 10.56.101 ARM and the No Child Left Behind Act, 
validation of full participation is required. Please complete this form as you prepare your materials for 
scoring. If you have any questions, please contact Judy Snow at 406-444-3656 or isnow(a)state.mt.us . 

Fax completed forms to Maria Bates at the Office of Public Instruction, 406-444-1369, by April 7, 2003. 

This form consists of three parts. There is a separate form for each of the grades (4, 8, and 1 1 ) that you 
are testing. 

Part I Enrollment update 

Part II Participation validation 

Part III Authorized signature 



PART I - Enrollment Update 



1 . February 2003 Grade «Grade» Reported 


1 . «Total» 


Enrollment 





2a. Enter the first day of testing at your school (a date 
within the March 10-28 testing window). 

2b. Enter the TOTAL number of grade «Grade» students 
enrolled* on the first day of testing. 



3. If the two enrollment numbers are different, please 


3. 


explain. If necessary, please attach documentation. 




*NOTE: Enrolled students include even those who 




attend only part of the school day. All enrolled 




students must be assessed. 
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PART II - Participation Validation 



1 . Number of grade «Grade» public school students 
who participated fully in the 2003 statewide 
assessment. This number includes students who 
participated in all areas being assessed by the Iowa 
Tests (with or without accommodations), the 
alternate assessment, or a combination of the two. 


1 . 




Grade «Grade» Participants 


2. Number of grade «Grade» public school students 
who were ABSENT for the ENTIRE testing period 
and were unable to participate in any part of the test, 
including the makeup. An answer document must be 
submitted for each of these students. 


2. 




Grade «Grade» Absentees 


3. Number of grade «Grade» public school students 
who completed some, but not all, areas of the 2003 
statewide assessment. 


3. 




Grade «Grade» Partial Testers 


4. Add the numbers in boxes 1 ,2, and 3 above. This 
total should equal the number in Part 1, box 2b. 


4. 




Total of boxes 1 , 2, and 3 


5. If Part II, box 4 does not match Part 1, box 2b, please 
explain. If necessary, please attach documentation. 


5. 



PART III - Authorized Signature 

I certify that: 

• an answer document has been submitted for every student enrolled; 

• the information provided on this form is correct to the best of my knowledge; 

• testing was completed in accordance with the directions in the “Montana Guide for Test 
Coordinators & Administrators 2002”, the update for 2003, and The Iowa Test Directions for 
Administration. 



District Name 


«District_Name» 


School Name 


«School_Name» School Code «SC» 


Name: (Please print) 


Title: (Principal or Supervising Teacher) 



Signature 



Date 



71 













CONSOLIDATED STATE APPLICATION ACCOUNTABILITY WORKBOOK 



Dear Linda, 

I spoke with Ron Tomalis, our acting Assistant Secretary, and he has approved your n-size of 20 without 
a confidence interval to make your AYP determinations this year. I will attach this email chain as an 
addendum to your approved state accountability plan indicating the use of 20 without a confidence 
interval this year and the use of 10 with a confidence interval in all future years. Please let me know if we 
can be of any further assistance to you all. 

Best, 

Celia 

Original Message 

From: McCulloch, Linda [ mailto:lmcculloch@state.mt.us ] 

Sent: Monday, August 04, 2003 4:58 PM 
To: 'celia.sims@ed.gov' 

Cc: Granbery, B J; Coopersmith, Nancy 
Subject: Change to Montana Plan 

Hi Celia, As you've been discussing with BJ Granbery of our office, we 
would like to use the minimum "N" of 20 this year instead of 10. In future 
years we will use a confidence interval in connection with the minimum of 
10. The discussion thus far with you has been that using 20 without a 
confidence interval would be roughly equivalent to 10 with a confidence 
interval. We hope you can approve right away and let us know so we can 
finish running our data. 

Thanks so much! 

***Linda McCulloch 

State Superintendent 

Montana Office of Public Instruction 

1227 11th Avenue 

PO Box 202501 

Helena, MT 59620-2501 



72 



